Self-supervised deep learning de-mosaic method for snapshot multi-band cameras

Abstract

This invention discloses a self-supervised deep learning demosaicing method suitable for snapshot-type multi-band cameras, using real mosaic images captured by the target camera for training. The method consists of two parts: self-supervised training and a two-branch model. Based on observation consistency constraints, the demosaicing model is reused, pseudo-paired data is generated, and the model is constrained to demosaic. Simultaneously, a robust interpolation algorithm is used as the low-frequency principal components of the model to avoid extreme model collapse, stabilizing network training and ultimately obtaining a demosaicing model that performs excellently on real data. The demosaicing model of this invention can be directly applied to real images without distortion or artifacts.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a self-supervised deep learning method for demosaicing. Background Technology

[0002] People often want to capture multiple pieces of light information, such as spectral and polarization information, in a single shot, so that it can be used in subsequent tasks such as pesticide detection, target tracking, and 3D reconstruction. One common method is to cover the camera's imaging focal plane with a periodically changing array of microfilters. Figure 1 As shown, each raw image captured in this way contains multi-band information such as spectral or polarization information, and the information from different bands is periodically mosaicked in space according to the filter arrangement pattern, which is generally called a mosaic image. Although this method is efficient and fast, it results in observational gaps in each band of the image, affecting the implementation of subsequent applications. Therefore, it is necessary to use a de-mosaic algorithm to recover the missing information of each band.

[0003] Deep learning-based desamicing methods are among the best performing existing approaches. However, most of these methods utilize image databases from other cameras to simulate mosaic images captured by snapshot-style multi-band cameras, creating paired data for supervised training of the network. However, the band correlations between simulated and real data differ, causing distortion in the trained desamicing deep network on real images. Furthermore, obtaining high-quality image databases is extremely difficult for some novel snapshot-style multi-band cameras, such as multispectral cameras operating in the infrared band. Therefore, a self-supervised deep learning desamicing method that does not rely on simulated data is urgently needed to effectively improve the desamicing performance of real images. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a self-supervised deep learning de-mosaicing method suitable for snapshot-type multi-band cameras, using real mosaic images captured by the target camera for training. The method consists of two parts: self-supervised training and a two-branch model. Based on observation consistency constraints, the de-mosaicing model is reused, pseudo-paired data is generated, and the model is constrained to de-mosaic. Simultaneously, a robust interpolation algorithm is used as the low-frequency principal components of the model to avoid extreme model collapse, stabilizing network training and ultimately obtaining a de-mosaicing model that performs excellently on real data. The de-mosaicing model of this invention can be directly applied to real images without distortion or artifacts.

[0005] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0006] Step 1: Collect training data;

[0007] A series of mosaic images captured by a snapshot multi-band camera were used as subsequent self-supervised training data.

[0008] Step 2: Build a demosaic model; use a deep network model as the backbone of the demosaic model, and set parameters to allow the mosaic image as input to the backbone; at the same time, build an interpolation branch that uses an interpolation algorithm. This branch interpolates the low-frequency components of the multi-band image by interpolating the sparse bands of the mosaic image. Finally, add the outputs of the backbone and the branch as the model output.

[0009] Step 3: Self-supervised training;

[0010] The training data is input into the demosaic model established in step 2 to obtain the initial multiband image cube, which is called multiband image No. 1.

[0011] The first multi-band image is randomly translated, flipped, and scaled to obtain a new image cube, which is called the second multi-band image. Then, the second multi-band image is input into the de-mosaic model again to obtain a new image cube, which is called the third multi-band image.

[0012] The difference between multi-band image No. 3 and multi-band image No. 2 is calculated. At the same time, the difference between multi-band image No. 1 after mosaic sampling and the original mosaic image is calculated. By minimizing the weighted sum of these two differences, the training process of the de-mosaic model is constrained, and the de-mosaic result, i.e., the multi-band image cube without missing parts, is output.

[0013] Preferably, step 3 is as follows:

[0014] Step 3-1: The sampling process of the mosaic image is considered as the multiplication and superposition of an ideal, defect-free multi-band image cube X and a periodic binary mask, that is:

[0015]

[0016] in The image represents the b-th band of X, where H, W, and B represent the image length, width, and number of bands, respectively, and M... b Y represents the binary periodic mask corresponding to the b-band during the imaging process, and Y represents the mosaic image generated by sampling.

[0017] Step 3-2: Input the mosaic image Y into the demosaic model to obtain the model's first output cube. It is called Multiband Image No. 1:

[0018]

[0019] Where θ is the parameter to be learned, and F(Y,θ) is the demosaic model;

[0020] Step 3-3: Randomly translate, flip, and scale the multi-band image No. 1, with the translation operation limited to one filter arrangement cycle, to obtain the multi-band image No. 2.

[0021]

[0022] Steps 3-4: Transform the resulting multi-band image No. 2. Using this as a pseudo-real value label, perform mosaic sampling on it to obtain a new pseudo-mosaic image.

[0023]

[0024] in It is image No. 2 (multi-band). The b-th band;

[0025] Obtain a paired image of a mosaic and a cube without missing parts: Then Inputting the depixelated model again yielded a second output cube.

[0026]

[0027] Steps 3-5: Use a loss function constraint and

[0028]

[0029] Where l(·) represents either L1 loss or L2 loss;

[0030] Steps 3-6: Use self-supervised mosaic loss to ensure the stability of the No. 1 multi-band image cube. The consistency between the mosaic image obtained by directly performing mosaic sampling without any transformation and the original input mosaic image:

[0031]

[0032] Steps 3-7: The overall loss function is as follows:

[0033]

[0034] Preferably, the interpolation algorithm is bilinear interpolation.

[0035] The beneficial effects of this invention are as follows:

[0036] Traditional supervised training methods using simulated mosaic images struggle to characterize the correlations between multiple bands of a real target camera, leading to distortion and artifacts in the demosaic results. Therefore, this invention proposes a self-supervised training method and corresponding network framework. The model output during training is used as a ground truth cube to create paired data, constraining the model for further optimization. This enables training directly using real images without the need for simulated data, allowing the trained demosaic model to be directly applied to real images without distortion or artifacts. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of a periodically changing microfilter array.

[0038] Figure 2(a) and Figure 2(b) show the results of supervised training of the 25-band demosaicing methods MCAN and InNet on real mosaic "leaves" images, respectively.

[0039] Figure 3 This is a schematic diagram of the mosaic sampling method and demosaic process of the present invention.

[0040] Figure 4 This invention is a self-supervised demosaic training framework.

[0041] Figure 5 This is a schematic diagram of the self-supervised demosaic model of the present invention.

[0042] Figure 6 This is a 25-band mosaic image library that has been self-supervised trained according to an embodiment of the present invention.

[0043] Figure 7 This is a visual comparison of the various unsupervised methods WB, ItSD, and PPID in the embodiments of the present invention with the present invention on test images.

[0044] Figure 8 This is a visual comparison of the supervised methods SpNet, MCAN, and InNet in the embodiments of the present invention with the present invention on the test images. Detailed Implementation

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0046] The purpose of this invention is to provide a self-supervised deep learning de-mosaic method suitable for snapshot multi-band cameras, so as to solve the de-mosaic problem of snapshot multi-band cameras in real-world situations.

[0047] A self-supervised deep learning de-mosaic method suitable for snapshot-type multi-band cameras includes the following steps:

[0048] Step 1: Collect training data. Use a snapshot multi-band camera to directly capture a series of scenes with rich spatial textures and varied content. The resulting mosaic images will be used as subsequent self-supervised training data.

[0049] Step 2: Build the demosaic model. A deep network is built using convolutional layers and non-linear activation layers as the model backbone. By setting relevant parameters, the backbone allows the mosaic image to be used as input. Simultaneously, an interpolation branch is built. This branch interpolates the sparse bands of the mosaic image to obtain stable low-frequency components of the multi-band image. Finally, the outputs of the backbone and the branch are added together to obtain the model output.

[0050] Step 3: Self-supervised training. Input the training data into the deep model established in Step 2 to obtain an initial multi-band image cube, called Multi-band Image 1. Perform some transformations such as translation, flipping, and scaling on this cube to obtain a new image cube, called Multi-band Image 2. Then sample this new image as a new mosaic image. Input the new mosaic image into the deep model again to obtain a new image cube, called Multi-band Image 3. Calculate the difference between Multi-band Image 3 and Multi-band Image 2. At the same time, calculate the difference between Multi-band Image 1 after mosaic sampling and the original mosaic image. By minimizing these two differences, constrain the optimization of the deep model. After training is complete, the de-mosaic result, i.e., the multi-band image cube without missing pixels, can be output. Specific implementation examples:

[0052] Snapshot multiband cameras, as an extremely compact and lightweight multiband imaging method, can acquire multiband information in a very short time and have no moving parts or complex optical paths, showing great promise for future applications. However, the spatial resolution and band resolution of the mosaic image obtained by this imaging method are mutually constrained, requiring demosaicing to recover the complete multiband image. As the band resolution increases, more spatial information is missing from each band, making demosaic algorithms increasingly reliant on band correlation. However, existing deep learning-based demosaic algorithms use simulated images for supervised training, and these simulated images are difficult to characterize the band correlation of real cameras. This leads to periodic distortions or artifacts in real images in the trained deep models, as shown in Figures 2(a) and (b). The supervised training methods MCAN and InNet either produce global periodic distortion or distortion at the edges on a real 25-band mosaic "leaf" image. To address these issues, this invention proposes a method that directly utilizes real images for self-supervised training of the demosaic network and proposes an applicable network framework for stable training, enabling it to perform excellently on real images.

[0053] Considering that conventional microfilter arrays are periodically arranged, the sampling process of a mosaic image can be viewed as the multiplication and superposition of an ideal cube X and a periodic binary mask, i.e.:

[0054]

[0055] in The image represents the b-th band of cube X, where H, W, and B represent the length, width, and number of bands of the image, respectively, and M... b The binary periodic mask corresponding to the b-band during imaging is represented by Y, which represents the mosaic image generated by sampling. This can be combined with... Figure 3 To further understand the mosaic sampling process.

[0056] Existing deep learning methods often utilize simulated databases to create paired X and Y pairs, and then perform supervised training on the deep network. The optimization formula can be expressed as:

[0057]

[0058] Where θ is the parameter to be learned by the deep network, F(Y,θ) is the network output, and l(·) is the pixel-level loss function used. The network trained in this way performs well on simulated images, but is prone to distortion and artifacts on real images.

[0059] Considering the demosaic degradation method, i.e., the periodic binary mask M, is known, if the current depth model can perfectly demosaic, then after transforming the multi-band cube output by the model through translation, flipping, etc., and resampling to obtain a new mosaic image, and inputting this image into the depth demosaic model, its output should also be consistent with the previously transformed multi-band cube. Therefore, a self-supervised demosaic training framework that only requires the mosaic image is proposed. The overall process is as follows: Figure 4 As shown.

[0060] First, the real mosaic image Y is fed into the demosaic depth model to obtain the model's first output cube. It is called Cube No. 1:

[0061]

[0062] Then, cube number 1 is randomly translated, flipped, rotated, or scaled to obtain the transformed cube number 2.

[0063]

[0064] Given that mosaic sampling is periodic, for the translation operation, the cube is randomly translated only within one arrangement period of the filters. The transformed, missing-value cube is used as a pseudo-ground value label, and mosaic sampling is performed on it to obtain a new pseudo-mosaic image.

[0065]

[0066] At this point, a paired image of a mosaic and a cube without missing parts is obtained: Then Inputting the previously used de-mosaic depth model, we obtained the second output of cube number 3 from the model.

[0067]

[0068] hope and To keep them as close as possible, a loss function is used to constrain them:

[0069]

[0070] Here, l(·) can use either the commonly used L1 or L2 loss. Also, based on the characteristics of the demosaic problem—that is, without any transformation of the first output cube (1) of the network, the resulting mosaic image obtained by directly performing mosaic sampling should be identical to the original input mosaic—were determined to be similar to the original input mosaic. Figure 1 Therefore, a self-supervised mosaic loss is proposed to guarantee this consistency:

[0071]

[0072] Here, l(·) is not restricted; either the commonly used L1 or L2 loss function can be used. Therefore, the total loss function is as follows:

[0073]

[0074] As can be seen from the above formula, there is an extreme case of model collapse: when the model output is simply a copy and stitching of the input mosaic image in the band dimension, the loss will be close to 0, but it does not actually perform any de-mosaic function. Therefore, an interpolation branch is added. This branch interpolates the sparse bands of the mosaic image to obtain the stable low-frequency components of the multi-band image. The interpolated multi-band image is added to the output of the depth model branch as the new model output, such as... Figure 5 As shown, the deep model branch only needs to predict the remaining high-frequency components, thus avoiding model collapse. The interpolation algorithm is not restricted here; simple bilinear interpolation will suffice.

[0075] By using the loss function of this invention in conjunction with stochastic gradient descent to progressively optimize the parameters in the deep model, a high-performing demosaic model can be obtained.

[0076] The camera used in this embodiment is a 25-band snapshot-type spectral multi-band camera, where each microfilter corresponds to one pixel, and the 25 filters repeat periodically in a 5×5 format. A series of mosaic images were captured using this camera to create an image library, prioritizing scenes with rich and varied lighting, colors, and textures, and cropping out the background areas. Figure 6 As shown, 30% of the images were used as test images, and the remainder were used as self-supervised training images. Since these images are all real images and do not have corresponding ground truth values ​​for the multi-band cube, a no-reference evaluation metric was used as the test metric.

[0077] This invention does not restrict the depth model of the main branch; any mainstream demosaic model is acceptable. The parameters of the input and output layers are modified according to the attributes of the 5×5 period and 25 bands.

[0078] The loss function l(·) uses the common L1 loss. The network is trained using 100×100 patches with a learning rate of 1×10⁻⁶. -4 The batch size was set to 1, and the number of iterations was set to 7300. Table 1 shows a comparison of metrics with other demosaic algorithms that do not require ground truth images, such as WB, ItSD, and PPID. A band from a 25-band image is used for visualization and comparison. Figure 7 As shown, this invention is significantly superior to other methods in both quantitative and qualitative comparisons, exhibiting no distortion or periodic missingness in the image space. This invention has also been compared with other supervised training deep model methods such as SpNet, MCAN, and InNet, as... Figure 8 As shown, compared with supervised deep learning methods, it is spatially sharp and free from periodic distortion, and there is no color distortion, meaning it also has good spectral fidelity.

[0079] Table 1 compares the unsupervised evaluation metrics of this invention with three other typical unsupervised methods on real test datasets.

[0080]

Claims

1. A self-supervised deep learning de-mosaic method suitable for snapshot-type multi-band cameras, characterized in that, Includes the following steps: Step 1: Collect training data; A series of mosaic images captured by a snapshot multi-band camera were used as subsequent self-supervised training data. Step 2: Build a demosaic model; use a deep network model as the backbone of the demosaic model, and set parameters to allow the mosaic image as input to the backbone; at the same time, build an interpolation branch that uses an interpolation algorithm. This branch interpolates the low-frequency components of the multi-band image by interpolating the sparse bands of the mosaic image. Finally, add the outputs of the backbone and the branch as the model output. Step 3: Self-supervised training; The training data is input into the demosaic model established in step 2 to obtain the initial multiband image cube, which is called multiband image No.

1. The first multi-band image is randomly translated, flipped, and scaled to obtain a new image cube, which is called the second multi-band image. Then, the second multi-band image is input into the de-mosaic model again to obtain a new image cube, which is called the third multi-band image. The difference between multi-band image No. 3 and multi-band image No. 2 is calculated. At the same time, the difference between multi-band image No. 1 after mosaic sampling and the original mosaic image is calculated. By minimizing the weighted sum of these two differences, the training process of the de-mosaic model is constrained, and the de-mosaic result, i.e., the multi-band image cube without missing parts, is output.

2. The self-supervised deep learning de-mosaic method for snapshot multi-band cameras according to claim 1, characterized in that, Step 3 is described in detail below: Step 3-1: The sampling process of the mosaic image is considered as the multiplication and superposition of an ideal, missing-free multi-band image cube X and a periodic binary mask, that is: in The image represents the b-th band of X, where H, W, and B represent the image length, width, and number of bands, respectively, and M... b Y represents the binary periodic mask corresponding to the b-band during the imaging process, and Y represents the mosaic image generated by sampling. Step 3-2: Input the mosaic image Y into the demosaic model to obtain the model's first output cube. It is called Multiband Image No. 1: Where θ is the parameter to be learned, and F(Y,θ) is the demosaic model; Step 3-3: Randomly translate, flip, and scale the multi-band image No. 1, with the translation operation limited to one filter arrangement cycle, to obtain the multi-band image No.

2. Steps 3-4: Transform the resulting multi-band image No.

2. Using this as a pseudo-real value label, perform mosaic sampling on it to obtain a new pseudo-mosaic image. in It is image No. 2 (multi-band). The b-th band; Obtain a paired image of a mosaic and a cube without missing parts: Then Inputting the depixelated model again yielded a second output cube. Steps 3-5: Use a loss function constraint and Where l(·) represents either L1 loss or L2 loss; Steps 3-6: Use self-supervised mosaic loss to ensure the stability of the No. 1 multi-band image cube. The consistency between the mosaic image obtained by directly performing mosaic sampling without any transformation and the original input mosaic image: Steps 3-7: The overall loss function is as follows:

3. The self-supervised deep learning de-mosaic method for snapshot multi-band cameras according to claim 1, characterized in that, The interpolation algorithm is bilinear interpolation.

Images