Method for faint light image denoising for multi-aperture camera

Abstract

The present application relates to a kind of faint light image denoising method for multi-aperture camera, comprising: step S1, utilizes multi-aperture camera to obtain N target scene images S of same time n ; Step S2, target scene image S n Based on registration matrix, registration is carried out, and the image S' n After registration is obtained;Step S3, construct multi-frame faint light image denoising network, and training is carried out;Step S4, the N images S' n After registration obtained in step S2 is sent to the network after training in step S3 and carries out inference.The present application can remove the noise of the image obtained by enhanced CCD, improve the signal-to-noise ratio of image, and is conducive to further improving the performance of faint light detector imaging, and widening its application field.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of optical remote sensing satellite information processing, and in particular to a faint light image denoising method for multi-aperture cameras. BACKGROUND

[0002] Remote sensing satellite platforms have the advantages of all-weather, high mobility, wide field of view, etc. However, when the light condition is weak, such as the dawn and dusk or night condition, the image effect obtained by visible light remote sensing is poor. With the development of faint light night vision technology, it is also possible to realize global night imaging by using night light remote sensing load. Faint light, also called night sky light, is the general term for weak "visible" light such as moonlight, starlight and atmospheric glow existing at night. Faint light night vision technology is an optoelectronic technology for studying the enhancement, transmission, storage, reproduction and application of images collected under low light conditions, and is an important part of modern optoelectronic technology. Due to the inherent characteristics of the human eye, when the ambient illuminance is relatively low, the human eye can only observe the outline of the object and cannot accurately identify the detailed features. Intensified CCD / CMOS (ICCD / ICMOS) is a widely used solid-state faint light imaging device that can work under very low illuminance conditions, which is coupled by an image intensifier and a CCD / CMOS. Although ICMOS can image under faint light night vision conditions, the image intensifier amplifies the noise intensity while enhancing the signal, resulting in a significant random noise in the acquired image, and the noise characteristics are more complex than the noise of traditional CMOS imaging.

[0003] Image denoising is a hot issue in the field of computer vision. Existing denoising algorithms can be divided into the following categories: spatial domain denoising algorithm, transform domain denoising algorithm, sparse representation based denoising algorithm, and deep learning based denoising algorithm. Spatial domain denoising algorithm mainly aims at the characteristic that natural image noise is independent and identically distributed in space, and uses filtering to remove noise; transform domain denoising algorithm first performs a specific transformation on the noisy image, then processes the transformed coefficients in the transform domain according to the characteristics of the transform domain and the properties of the noise, removes the noise components and retains the signal components; the sparse representation method is to represent the noisy image by a certain overcomplete atomic library, to represent the original signal by a number of large signals, and to separate the image and the noise by using sparsity; the deep learning method learns the distribution characteristics of the noise from a large number of noise data samples, so as to separate the noise and the image signal.

[0004] The micro-light image, in particular the image acquired by the IC-MOS detector, has great difference from the natural image acquired under normal illumination, the signal-to-noise ratio is low, the random noise is obvious, and the overall visual effect of the image is greatly reduced, which is not conducive to the observation and identification of the human eye, and the multi-aperture camera has certain multi-frame advantages, how to utilize the advantages to improve the quality of the micro-light image becomes a problem to be solved. SUMMARY

[0005] To solve the technical problems existing in the prior art, the purpose of the present application is to provide a micro-light image denoising method for a multi-aperture camera, which can improve cloud removal effect, reduce cloud residue and reduce cloud and fog obstruction, and greatly improve the usability of remote sensing data.

[0006] To achieve the above-mentioned purpose of the application, the present application provides a micro-light image denoising method for a multi-aperture camera, comprising the following steps:

[0007] Step S1, acquiring N images of a target scene at the same time by using a multi-aperture camera n ;

[0008] Step S2, registering the target scene images S n based on a registration matrix to obtain registered images S' n ;

[0009] Step S3, constructing a multi-frame micro-light image denoising network and training the network;

[0010] Step S4, sending the N registered images S' n obtained in step S2 into the network trained in step S3 for inference.

[0011] According to one technical solution of the present application, in step S1, the multi-aperture camera is arranged in a "one" shape.

[0012] According to one technical solution of the present application, before step S2, the pixel misalignment between the sub-apertures of the multi-aperture camera is initially corrected, a target is shot, and a registration method is used to take one of the images as a reference frame to register five target images to obtain a registration matrix M j , j = 1, …, 4.

[0013] According to one technical solution of the present application, in step S2, the ORB method is used, the registration matrix M j is used on the target scene images S n to obtain the registered images S n '.

[0014] According to one technical solution of the present application, in step S3, it specifically comprises:

[0015] Step S31, constructing a data set including micro-light noise data and true value data;

[0016] Step S32, constructing a multi-aperture micro-light image denoising network;

[0017] Step S33, inputting the data set into the multi-aperture micro-light image denoising network to obtain a trained multi-aperture micro-light image denoising network.

[0018] According to one technical solution of the present application, in the step S31, the micro-light noise data is jointly constituted by images and micro-light data sets taken in a multi-aperture micro-light camera laboratory, and the true value data is obtained by superimposing a plurality of continuously taken images.

[0019] According to one technical solution of the present application, in the step S32, the multi-aperture micro-light image denoising network comprises an input module, a feature extraction module, a fusion module, and an output module.

[0020] The input module comprises N input sub-modules, each input sub-module comprising a convolution layer and a ReLu activation unit, which maps a single-channel image to a higher-dimensional feature space, and then enters the N feature extraction sub-modules to extract the features of each image;

[0021] The feature extraction module comprises N feature extraction sub-modules, the number of feature maps of each feature extraction sub-module gradually decreases, and each feature extraction sub-module is composed of a residual and a convolution layer, the feature extraction module outputs the denoised image feature map of each image, and the N feature maps enter the fusion module;

[0022] The fusion module introduces a channel attention mechanism to learn the feature weight distribution between each image, the feature map input by each image is subjected to maximum pooling and mean pooling to generate two-dimensional vectors, which are added after linear optimization by a fully connected layer to obtain a channel attention vector of the input feature map, and finally multiplied by the input feature map to obtain the output feature map of the fusion module;

[0023] The output feature map is mapped to a denoised image with the same input dimension by the output module.

[0024] According to one aspect of the present application, an electronic device is provided, comprising one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected with the memory, and the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes a multi-aperture camera micro-light image denoising method according to any one of the above technical solutions.

[0025] According to one aspect of the present application, a computer readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement a multi-aperture camera oriented faint light image denoising method as described in any of the above technical solutions.

[0026] Compared with the prior art, the present application has the following beneficial effects:

[0027] The present application provides a multi-aperture camera oriented faint light image denoising method, which takes advantage of the multi-aperture camera to simultaneously acquire multiple frames of image information of the same scene / target at the same time, extracts additional information generated by sub-pixel level misalignment, and, by referring to traditional multi-frame superposition denoising methods and combining deep learning techniques, feeds the registered multiple frames of image into a constructed deep learning network to realize faint light image denoising and improve image signal-to-noise ratio, which is conducive to further improving the performance of faint light detector imaging and broadening its application field. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, brief introductions will be given below to the drawings needed in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0029] Figure 1 A flowchart of a multi-aperture camera oriented faint light image denoising method in an embodiment of the present application is schematically shown;

[0030] Figure 2 A flowchart of a multi-aperture camera oriented faint light image denoising method in another embodiment of the present application is schematically shown;

[0031] Figure 3 A structural schematic diagram of a multi-aperture camera used in an embodiment of the present application is schematically shown;

[0032] Figure 4 A schematic diagram of a multi-aperture denoising neural network training process used in an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0033] The description of the embodiments of the present application should be combined with the corresponding drawings, which should be considered as a part of the complete description. In the drawings, the shapes or thicknesses of the embodiments can be exaggerated and simplified or convenient for illustration. Moreover, parts of the structures in the drawings will be described separately, and it should be noted that the elements not shown or not described in the drawings are in the form known to those skilled in the art.

[0034] The description of the embodiments herein, any reference to direction or position, is only for the convenience of description and cannot be understood as any limitation on the scope of protection of the present application. The following description of the preferred embodiments will involve combinations of features that can exist independently or in combination, and the present application is not particularly limited to the preferred embodiments. The scope of the present application is defined by the claims.

[0035] As shown in Figure 1 , a method for faint light image denoising of a multi-aperture camera according to the present application comprises the following steps:

[0036] Step S1, acquiring N images of the target scene at the same time using a multi-aperture camera n ;

[0037] In an embodiment, N = 5, the sub-apertures of the multi-aperture camera are arranged in a line, as shown in Figure 3 , five images S1, S2, S3, S4, and S5 are acquired respectively.

[0038] Step S2, registering the target scene image S n based on the registration matrix to obtain the registered image S' n ;

[0039] Before shooting the images, the pixel misalignment between the sub-apertures of the multi-aperture camera is corrected initially, by shooting a target, using a registration method to take one of the images as a reference frame, registering the five target images to obtain the registration matrix M j , j = 1, …, 4. Using the ORB (Oriented FAST and Rotated BRIEF) method, the registration matrix M j is introduced into step S1 to register the target scene image S n to obtain the registered image S n '

[0040] Step S3, constructing a multi-frame faint light image denoising network and training, specifically including:

[0041] Step S31, constructing a data set including faint light noise data and true value data;

[0042] The faint light noise data is composed of images and faint light data sets shot in the multi-aperture faint light camera laboratory, and the true value data is obtained by superimposing multiple images shot continuously.

[0043] Step S32, constructing a multi-aperture faint light image denoising network, as shown in Figure 4 ;

[0044] The multi-aperture micro-light image denoising network comprises an input module, a feature extraction module, a fusion module, and an output module.

[0045] The input module comprises five input sub-modules, each of which comprises a convolution layer and a ReLu activation unit, and maps a single-channel image to a higher-dimensional feature space, and then enters the five feature extraction sub-modules to extract the features of each image.

[0046] The feature extraction module comprises five feature extraction sub-modules, each of which extracts the features of each image, and the number of feature maps of each feature extraction sub-module gradually decreases, and each feature extraction sub-module comprises a residual and a convolution layer, and the mapping function of the residual is:

[0047] RES(x)=ReLU(f(x)+x)

[0048] where RES(x) represents the mapping function of the entire residual, x represents the feature map input of the residual, ReLU represents a nonlinear activation unit, f(x) represents the output result of the intermediate layer, and the residual is realized through a jump connection, which can reduce the difficulty of network training and solve problems such as gradient degradation. The feature map dimensions of each feature extraction sub-module are 256, 128, 64, and 32 from front to back, the convolution layer is a 3×3 convolution kernel, the feature extraction module outputs the denoised image feature map of each image, and the five feature maps enter the fusion module.

[0049] The fusion module introduces a channel attention mechanism, performs global maximum pooling and global average pooling on the input feature map in the spatial dimension, obtains two 1×1×5C feature maps, where C is the dimension of a single-channel feature map, then sends the results of the global maximum pooling and the global average pooling into a multilayer perceptron for learning, finally fuses and maps the output results of the multilayer perceptron through an activation function, and finally obtains a channel attention weight matrix M c ,

[0050] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)))

[0051] The weight matrix M c is multiplied by the input feature map to obtain the output feature map of the fusion module.

[0052] The output feature map passes through the output module and is mapped into a denoised image with the same input dimension.

[0053] The output feature map passes through the output module and is mapped into a denoised image with the same input dimension.

[0054] Step S33: Input the dataset into the multi-aperture low-light image denoising network to obtain the trained multi-aperture low-light image denoising network.

[0055] Step S4: Register the N images S' obtained in step S2 n The image is fed into the network trained in step S3 for inference, resulting in a denoised image, thus achieving denoising of multi-aperture low-light images.

[0056] According to one aspect of the present invention, an electronic device is provided, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory; when the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform a low-light image denoising method for multi-aperture cameras as described in any of the above technical solutions.

[0057] According to one aspect of the present invention, a computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement a low-light image denoising method for a multi-aperture camera as described in any of the above technical solutions.

[0058] Computer-readable storage media can include any medium capable of storing or transmitting information. Examples of computer-readable storage media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and so on. Code segments can be downloaded via computer networks such as the Internet and intranets.

[0059] The present invention provides a low-light image denoising method for multi-aperture cameras, comprising the following steps: Step S1, acquiring N target scene images S at the same time using a multi-aperture camera. n Step S2: Based on the registration matrix, perform registration on the target scene image S. n Registration is performed to obtain the registered image S' n Step S3: Construct a denoising network for multi-frame low-light images and train it; Step S4: Combine the registered N images S' obtained in Step S2 n The images are fed into the network trained in step S3 for inference. This invention leverages the advantage of multi-aperture cameras to simultaneously acquire multiple frames of the same scene / target image, obtaining multi-frame image information at the same time. It extracts additional information caused by sub-pixel level misalignment, draws on traditional multi-frame stacking denoising methods, and combines deep learning techniques to feed the registered multi-frame images into the constructed deep learning network. This achieves denoising of low-light images, improves the image signal-to-noise ratio, and is beneficial for further enhancing the imaging performance of low-light detectors and broadening their application areas.

[0060] Furthermore, it should be noted that the present invention can be provided as a method, apparatus, or computer program product. Therefore, embodiments of the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0061] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0062] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0063] It should also be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0064] Finally, it should be noted that the above description is of preferred embodiments of the application, and that although preferred embodiments of the application have been described, numerous changes and modifications can be made to the preferred embodiments without departing from the principles of the application, and that these changes and modifications are intended to be included within the scope of the application. Accordingly, the appended claims are intended to cover all such changes and modifications as fall within the scope of the application.

Claims

1. A faint light image denoising method for a multi-aperture camera, characterized in that, The method comprises the following steps: Step S1, acquiring N images of the target scene at the same time by using a multi-aperture camera S n The multi-aperture camera is arranged in a "one" shape. Before step S2, the initial correction of the pixel misalignment between the sub-apertures of the multi-aperture camera is performed. By shooting a target, the registration method is used to take one of the images as a reference frame to register the five target images, and the registration matrix between the sub-apertures is obtained ; Step S2, registering the target scene image based on the registration matrix S n Registration is performed to obtain a registered image, specifically including: adopting an ORB method, and applying the registration matrix to the target scene image to obtain the registered image M j the target scene image S n to obtain a registered image ; Step S3, constructing a multi-frame faint light image denoising network and training the same; The multi-aperture faint light image denoising network comprises an input module, a feature extraction module, a fusion module and an output module; The input module comprises N input sub-modules, each input sub-module comprising a convolution layer and a ReLu activation unit, which maps a single-channel image to a higher-dimensional feature space and then enters the N feature extraction sub-modules to extract features of each image; The feature extraction module comprises N feature extraction sub-modules, the number of feature maps of each feature extraction sub-module gradually decreases, each feature extraction sub-module comprises a residual and a convolution layer, the feature extraction module outputs a denoised image feature map of each image, and the N feature maps enter the fusion module; The fusion module introduces a channel attention mechanism to learn the feature weight distribution between each image, the feature map input by each image is subjected to maximum pooling and mean pooling to generate two-dimensional vectors, which are linearly optimized by a full connection layer and then added to obtain a channel attention vector of the input feature map, and finally multiplied by the input feature map to obtain an output feature map of the fusion module; The output feature map is mapped to a denoised image with the same input dimension by the output module; Step S4, the N images obtained in step S2 are registered The trained network is sent to step S3 for inference.

2. The multi-aperture camera oriented faint image denoising method according to claim 1, wherein, In step S3, the following steps are included: Step S31, constructing a data set comprising faint light noise data and true value data; Step S32, constructing a multi-aperture faint light image denoising network; Step S33, inputting the data set into the multi-aperture faint light image denoising network to obtain a trained multi-aperture faint light image denoising network.

3. The multi-aperture camera oriented low-light image denoising method of claim 2, wherein, In step S31, the faint light noise data is composed of images and faint light data sets taken in a multi-aperture faint light camera laboratory, and the true value data is obtained by superimposing a plurality of continuously taken images.

4. An electronic device, comprising: One or more processors, one or more memories, and one or more computer programs; wherein the processor is connected with the memory, the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes the multi-aperture camera-oriented faint light image denoising method according to any one of claims 1 to 3. A computer program for storing computer instructions, which are executed by a processor to implement the multi-aperture camera-oriented faint light image denoising method according to any one of claims 1 to 3.

5. A computer readable storage medium, characterized in that, ​

Images