Low-light image denoising method for multi-aperture camera

Abstract

The invention relates to a low-light-level image denoising method for a multi-aperture camera. The method comprises the following steps: S1, acquiring N target scene images Sn at the same moment by using the multi-aperture camera; s2, registering the target scene image Sn based on the registration matrix to obtain a registered image S'n; s3, constructing a de-noising network oriented to the multi-frame low-light image, and training the de-noising network; and S4, sending the registered N images S'n obtained in the step S2 into the network trained in the step S3 for reasoning. According to the invention, the noise of the image obtained by the enhanced CCD can be removed, the signal-to-noise ratio of the image is improved, the imaging performance of the low-light detector is further improved, and the application field of the low-light detector is widened.

Description

Technical Field

[0001] The present invention relates to the technical field of optical remote sensing satellite information processing, and in particular to a low-light-level image denoising method for a multi-aperture camera. Background Art

[0002] Remote sensing satellite platforms have advantages such as all-day, high mobility, and wide field of view. However, when the lighting conditions are weak, such as at dusk or at night, the image obtained by visible light remote sensing is not good. With the development of low-light-level night vision technology, night-light remote sensing payloads that can achieve global night imaging have also become possible. Low light, also known as night sky light, is a general term for weak "visible" light such as moonlight, starlight, and atmospheric glow that exist at night. Low-light-level night vision technology is an important part of modern optoelectronic technology to study the photoelectric technology of enhancing, transmitting, storing, reproducing and applying the collected images under low light conditions. Due to the inherent characteristics of the human eye, when the ambient light intensity is relatively low, the human eye can only observe the outline of the object and cannot accurately identify the detailed features. Intensified CCD / CMOS (ICCD / ICMOS) is a widely used solid-state low-light imaging device that can work under very low illumination conditions. It is composed of an image intensifier and a CCD / CMOS coupled. Although ICMOS can image under low-light night vision conditions, the image intensifier amplifies the intensity of noise while enhancing the signal, resulting in obvious random noise in the acquired image, and the noise characteristics are more complex than the noise of traditional CMOS imaging.

[0003] Image denoising is a hot topic in the field of computer vision. Existing denoising algorithms can be divided into the following categories: spatial domain denoising algorithm, transform domain denoising algorithm, denoising algorithm based on sparse representation, and denoising algorithm based on deep learning. The spatial domain denoising algorithm mainly uses filtering to remove noise based on the independent and identically distributed characteristics of natural image noise in space; the transform domain denoising algorithm first performs a specific transformation on the noisy image, and then processes the transform coefficients in the transform domain according to the characteristics of the transform domain and the nature of the noise, removes the noise component, and retains the signal component; the sparse representation method is to sparsely represent the noisy image through a certain overcomplete atomic library, use several large signals to represent the original signal, and use sparsity to separate the image from the noise; the deep learning method separates the noise and image signals by learning the distribution characteristics of the noise from a large number of noise data samples.

[0004] Low-light images, especially those acquired by ICMOS detectors, are quite different from natural images acquired under normal illumination. They have low signal-to-noise ratio and obvious random noise, which greatly reduces the overall image perception and is not conducive to human observation and recognition. Multi-aperture cameras have certain advantages of multiple frames. How to use this advantage to improve the quality of low-light images has become a problem that needs to be solved. Summary of the invention

[0005] In order to solve the technical problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a low-light image denoising method for multi-aperture cameras, which can improve the cloud removal effect, reduce residual cloud, reduce cloud and fog occlusion, and greatly improve the availability of remote sensing data.

[0006] To achieve the above-mentioned object of the invention, the present invention provides a low-light image denoising method for a multi-aperture camera, comprising the following steps:

[0007] Step S1: Use a multi-aperture camera to obtain N images S of the target scene at the same time. n ;

[0008] Step S2: Based on the registration matrix, the target scene image S n Perform registration to obtain the registered image S' n ;

[0009] Step S3, constructing a denoising network for multi-frame low-light images and performing training;

[0010] Step S4: The N images S' obtained in step S2 are registered. n The trained network in step S3 is fed into the network for inference.

[0011] According to a technical solution of the present invention, in step S1, the multiple spatial cameras are arranged in a line.

[0012] According to a technical solution of the present invention, before step S2, the pixel misalignment between the sub-apertures of the multi-aperture camera is initially corrected, and the five target images are registered by shooting the target and using a registration method to take one of the images as a reference frame to obtain a registration matrix M between the sub-apertures. j , j=1,…,4.

[0013] According to a technical solution of the present invention, in step S2, the ORB method is used to convert the registration matrix M j For the target scene image S n , get the registered image S n '.

[0014] According to a technical solution of the present invention, step S3 specifically includes:

[0015] Step S31, constructing a data set including low-light noise data and true value data;

[0016] Step S32, constructing a denoising network for multi-aperture low-light images;

[0017] Step S33: input the data set into the multi-aperture low-light-level image denoising network to obtain a trained multi-aperture low-light-level image denoising network.

[0018] According to a technical solution of the present invention, in step S31, the low-light noise data is composed of images taken by a multi-aperture low-light camera laboratory and a low-light data set, and the true value data is obtained by superimposing multiple frames of images taken continuously.

[0019] According to a technical solution of the present invention, in step S32, the multi-aperture low-light image denoising network includes an input module, a feature extraction module, a fusion module, and an output module;

[0020] The input module includes N input submodules, each of which includes a convolutional layer and a ReLu activation unit, maps a single-channel image to a higher-dimensional feature space, and then enters the N-channel feature extraction module to extract features of each image respectively;

[0021] The feature extraction module includes N feature extraction submodules, the number of feature maps of each feature extraction submodule gradually decreases, the feature extraction submodule is composed of residual and convolution layers, the feature extraction module outputs the image feature map after denoising of each image respectively, and the N feature maps enter the fusion module;

[0022] The fusion module introduces a channel attention mechanism to learn the feature weight distribution between each image. The feature map of each image input is subjected to maximum pooling and mean pooling to generate two-dimensional vectors, which are respectively added after linear optimization of the fully connected layer to obtain the channel attention vector of the input feature map, and finally multiplied with the input feature map to obtain the output feature map of the fusion module;

[0023] The output feature map is mapped into a denoised image with the same dimension as the input image through the output module.

[0024] According to one aspect of the present invention, an electronic device is provided, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes a low-light image denoising method for a multi-aperture camera as described in any one of the above technical solutions.

[0025] According to one aspect of the present invention, a computer-readable storage medium is provided for storing computer instructions. When the computer instructions are executed by a processor, a low-light image denoising method for a multi-aperture camera as described in any one of the above technical solutions is implemented.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The present invention proposes a low-light image denoising method for multi-aperture cameras. The method takes advantage of the multi-aperture camera's ability to simultaneously acquire multiple frames of images of the same scene / target, acquires multiple frames of image information at the same time, extracts additional information caused by sub-pixel misalignment, draws on traditional multi-frame superposition denoising methods, combines deep learning technology, and sends the registered multi-frame images to the constructed deep learning network to achieve low-light image denoising and improve image signal-to-noise ratio, which is beneficial to further improve the imaging performance of low-light detectors and broaden their application fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 A flowchart of a low-light image denoising method for a multi-aperture camera in an embodiment of the present invention is schematically shown;

[0030] Figure 2 A flowchart schematically showing a low-light image denoising method for a multi-aperture camera according to another embodiment of the present invention;

[0031] Figure 3 A schematic diagram schematically shows the structure of a multi-aperture camera used in an embodiment of the present invention;

[0032] Figure 4 The diagram schematically shows a multi-aperture denoising neural network training process in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The description of the embodiments of this specification should be combined with the corresponding drawings, which should be considered as part of the complete specification. In the drawings, the shape or thickness of the embodiments may be enlarged and indicated for simplification or convenience. Furthermore, the parts of each structure in the drawings will be described separately. It is worth noting that the elements not shown in the drawings or not described in words are in a form known to ordinary technicians in the relevant technical field.

[0034] The description of the embodiments herein and any reference to directions and orientations are only for the convenience of description and are not to be construed as any limitation on the scope of protection of the present invention. The following description of the preferred embodiments may involve combinations of features, which may exist independently or in combination, and the present invention is not particularly limited to the preferred embodiments. The scope of the present invention is defined by the claims.

[0035] like Figure 1 As shown, a low-light image denoising method for a multi-aperture camera of the present invention comprises the following steps:

[0036] Step S1: Use a multi-aperture camera to obtain N images S of the target scene at the same time. n ;

[0037] In the embodiment, N=5, and the sub-apertures of the multi-aperture camera are arranged in a row, such as Figure 3 As shown, five images S1, S2, S3, S4, and S5 are obtained respectively.

[0038] Step S2: Based on the registration matrix, the target scene image S n Perform registration to obtain the registered image S' n ;

[0039] Before shooting the image, the pixel misalignment between the sub-apertures of the multi-aperture camera is initially corrected. By shooting the target, the registration method is used to register the five target images with one of the images as the reference frame to obtain the registration matrix M between the sub-apertures. j , j = 1,…, 4. Using the ORB (Oriented FAST and Rotated BRIEF) method, the registration matrix M j In step S1, the target scene image S n Perform registration to obtain the registered image S n '

[0040] Step S3: construct a denoising network for multi-frame low-light images and perform training, specifically including:

[0041] Step S31, constructing a data set including low-light noise data and true value data;

[0042] The low-light noise data are composed of images taken by a multi-aperture low-light camera laboratory and a low-light data set, and the true value data are obtained by superimposing multiple frames of images taken continuously.

[0043] Step S32: construct a multi-aperture low-light image denoising network, such as Figure 4 As shown;

[0044] The multi-aperture low-light image denoising network comprises an input module, a feature extraction module, a fusion module, and an output module;

[0045] The input module includes 5 input submodules, each of which includes a convolutional layer and a ReLu activation unit, mapping a single-channel image to a higher-dimensional feature space, and then entering the 5-way feature extraction module to extract the features of each image respectively;

[0046] The feature extraction module includes 5 feature extraction submodules, which extract the features of each image respectively. The number of feature maps of each feature extraction submodule gradually decreases. The feature extraction submodule consists of residual and convolution layers. The mapping function of the residual is:

[0047] RES(x)=ReLU(f(x)+x)

[0048] Among them, RES(x) represents the mapping function of the entire residual, x represents the feature map of the residual input, ReLU represents the nonlinear activation unit, f(x) represents the output result of the intermediate layer, and the residual is realized through jump connection, which can reduce the difficulty of network training and solve problems such as gradient degradation. The feature map dimensions of each feature extraction submodule are 256, 128, 64, and 32 dimensions from front to back, and the convolution layer is a 3×3 convolution kernel. The feature extraction module outputs the image feature map after denoising for each image, and the 5-way feature map enters the fusion module;

[0049] The fusion module introduces the channel attention mechanism, performs global maximum pooling and global average pooling on the input feature map in the spatial dimension, and obtains two 1×1×5C feature maps, where C is the dimension of the single-channel feature map. Then, the results of global maximum pooling and global average pooling are respectively sent to the multi-layer perceptron for learning. Finally, the results of the multi-layer perceptron output are fused and mapped with the activation function to finally obtain the channel attention weight matrix M c ,

[0050] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)))

[0051] Weight matrix M c Multiply the input feature map to obtain the output feature map of the fusion module;

[0052] The output feature map is mapped to a denoised image with the same dimension as the input through the output module;

[0053] The output feature map is mapped into a denoised image with the same dimension as the input image through the output module.

[0054] Step S33: input the data set into the multi-aperture low-light-level image denoising network to obtain a trained multi-aperture low-light-level image denoising network.

[0055] Step S4: The N images S' obtained in step S2 are registered. n The trained network in step S3 is sent to reasoning to obtain a denoised image, thereby achieving denoising of multi-aperture low-light images.

[0056] According to one aspect of the present invention, an electronic device is provided, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes a low-light image denoising method for a multi-aperture camera as described in any of the above technical solutions.

[0057] According to one aspect of the present invention, a computer-readable storage medium is provided for storing computer instructions. When the computer instructions are executed by a processor, a low-light image denoising method for a multi-aperture camera as described in any one of the above technical solutions is implemented.

[0058] Computer-readable storage media may include any medium capable of storing or transmitting information. Examples of computer-readable storage media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. Code segments may be downloaded via computer networks such as the Internet, intranets, etc.

[0059] The present invention provides a low-light image denoising method for a multi-aperture camera, comprising the following steps: Step S1, using a multi-aperture camera to obtain N images S of a target scene at the same time. n ; Step S2, based on the registration matrix, the target scene image S n Perform registration to obtain the registered image S' n ; Step S3, construct a multi-frame low-light image denoising network and train it; Step S4, obtain the N images S' after registration in step S2 n The trained network in step S3 is sent to reasoning. The present invention uses the advantage of a multi-aperture camera to simultaneously obtain multiple frames of the same scene / target image, obtains multiple frames of image information at the same time, extracts additional information generated by sub-pixel level misalignment, draws on traditional multi-frame superposition denoising methods, combines deep learning technology means, and sends the registered multi-frame images to the constructed deep learning network to achieve denoising of low-light images and improve image signal-to-noise ratio, which is conducive to further improving the imaging performance of low-light detectors and broadening their application fields.

[0060] In addition, it should be noted that the present invention can be provided as a method, an apparatus or a computer program product. Therefore, the embodiments of the present invention can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0061] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0062] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0063] It should also be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0064] Finally, it should be noted that the above is a preferred embodiment of the present invention. It should be pointed out that although the preferred embodiment of the present invention has been described, for those skilled in the art, once the basic creative concept of the present invention is known, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention. Therefore, the attached claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the embodiments of the present invention.

Claims

1. A low-light image denoising method for multi-aperture cameras, characterized in that: The following steps are involved: Step S1: Use a multi-aperture camera to obtain N images S of the target scene at the same time. n ; Step S2: Based on the registration matrix, the target scene image S n Perform registration to obtain the registered image S' n ; Step S3, constructing a denoising network for multi-frame low-light images and performing training; Step S4: The N images S' obtained in step S2 are registered. n The trained network in step S3 is fed into the network for inference.

2. The low-light image denoising method for multi-aperture cameras according to claim 1, characterized in that: In step S1, the multiple space cameras are arranged in a straight line.

3. The low-light image denoising method for multi-aperture cameras according to claim 2, characterized in that: Before step S2, the pixel misalignment between the sub-apertures of the multi-aperture camera is initially corrected. By shooting the target, the five target images are registered using a registration method with one of the images as a reference frame to obtain a registration matrix M between the sub-apertures. j , j=1,…,4.

4. The low-light image denoising method for multi-aperture cameras according to claim 3, characterized in that: In step S2, the ORB method is used to convert the registration matrix M j For the target scene image S n , get the registered image S n '.

5. The low-light image denoising method for multi-aperture cameras according to claim 2, characterized in that: The step S3 specifically includes: Step S31, constructing a data set including low-light noise data and true value data; Step S32, constructing a denoising network for multi-aperture low-light images; Step S33: input the data set into the multi-aperture low-light-level image denoising network to obtain a trained multi-aperture low-light-level image denoising network.

6. The low-light image denoising method for multi-aperture cameras according to claim 5, characterized in that: In the step S31, the low-light noise data is composed of images taken by a multi-aperture low-light camera laboratory and a low-light data set, and the true value data is obtained by superimposing multiple frames of images taken continuously.

7. The low-light image denoising method for multi-aperture cameras according to claim 6, characterized in that: In the step S32, the multi-aperture low-light image denoising network comprises an input module, a feature extraction module, a fusion module, and an output module; The input module includes N input submodules, each of which includes a convolutional layer and a ReLu activation unit, maps a single-channel image to a higher-dimensional feature space, and then enters the N-channel feature extraction module to extract features of each image respectively; The feature extraction module includes N feature extraction submodules, the number of feature maps of each feature extraction submodule gradually decreases, the feature extraction submodule is composed of residual and convolution layers, the feature extraction module outputs the image feature map after denoising of each image respectively, and the N feature maps enter the fusion module; The fusion module introduces a channel attention mechanism to learn the feature weight distribution between each image. The feature map of each image input is subjected to maximum pooling and mean pooling to generate two-dimensional vectors, which are respectively added after linear optimization of the fully connected layer to obtain the channel attention vector of the input feature map, and finally multiplied with the input feature map to obtain the output feature map of the fusion module; The output feature map is mapped into a denoised image with the same dimension as the input image through the output module.

8. An electronic device, characterized in that: include: One or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes the low-light image denoising method for a multi-aperture camera as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, implement the low-light image denoising method for a multi-aperture camera according to any one of claims 1 to 7.

Images