Image moire removal method based on focus-virtual focus dual camera

Abstract

A method for removing moiré patterns from images based on a dual-camera setup with both focus and out-of-focus lenses includes the following steps: Step 1: Setting up the main camera and secondary camera side-by-side; Step 2: Using the RAFT optical flow method to align the focused image from the main camera and the out-of-focus image from the secondary camera to resolve differences in displacement and occlusion between the two images; Step 3: ... A and I D Input the demoiré network, and I D As a guide, let the network learn to remove moiré patterns. R Step 4: Recovery based on joint bilateral filtering. The advantages of this invention are: it uses a dual-camera system to acquire focused and out-of-focus images. The focused image contains high-quality texture but may exhibit moiré patterns, while the out-of-focus image has significantly reduced moiré patterns but some texture blur. By performing demoiré processing on both the focused and out-of-focus images, the moiré patterns in the image can be accurately and efficiently removed.

Description

Technical Field

[0001] This invention belongs to the field of image processing, specifically a method for removing moiré patterns from images based on a dual-camera system with both focus and defocus capabilities. Background Technology

[0002] Moiré patterns are a type of visual artifact that appears in images when the sampling frequency of a periodic light signal matches the sampling frequency of a digital camera. This has become a common problem in both everyday and professional photography, severely impacting image quality. However, as... Figure 1 As shown, in real-world scenarios, moiré patterns and real textures are very similar, and existing technologies make it difficult to distinguish between the two, resulting in moiré patterns or overly smoothed textures in the imaging results.

[0003] In existing moiré removal techniques, hardware solutions involve modifying the camera's optical components to reduce periodic variations in the light signal, thereby avoiding moiré patterns caused by discrete sampling of CMOS / CCD sensors. Examples include adding an anti-aliasing low-pass filter in front of the camera lens. However, these methods often result in image blurring and reduced overall image quality. In recent years, software solutions have also been developed. Traditional algorithms for single-image moiré removal include interpolation-based, filtering-based, and layer decomposition-based algorithms. However, while preserving image texture details, their effectiveness in removing large-scale moiré patterns is limited. Recently, deep learning-based algorithms have proven to be more effective. These algorithms utilize multi-scale networks (such as networks with UNet structures or encoder / decoder structures, global-to-local networks, etc.) and transform domain networks based on the frequency or wavelet domains. Furthermore, different algorithms employ different training loss functions, including original or modified L1 or L2 losses, VGG losses (for visual enhancement), adversarial losses and prior losses from discriminators, and high-frequency domain losses. Unlike single-image demoiring methods, video demoiring algorithms utilize additional information from adjacent frames to map the input image to a moiré-free image. This is achieved through multi-scale feature alignment, fusion, and demoiring. Similarly, multi-image demoiring algorithms take multiple frames as input and align them, while using a multi-scale feature encoding module to enhance low-frequency information. In this case, the focus-defocus demoiring algorithm indirectly uses a moiré-free out-of-focus image as a learning target during the training phase to aid in demoiring the input image. However, as the current problem states, moiré patterns in images are very similar to normal textures. Single-image demoiring algorithms often struggle to distinguish between moiré patterns and normal textures when processing single-shot images, easily resulting in color errors and blurred details in the processing results. This is one of the current challenges faced by image moiré removal tasks. Summary of the Invention

[0004] This invention proposes an image demoiring framework based on a focus-defocus dual-camera system to address the shortcomings of existing technologies.

[0005] This invention is achieved through the following technical solution:

[0006] The image moiré removal method based on a dual-camera system with both focus and defocus capabilities includes the following steps:

[0007] Step 1: Set up the main camera and the secondary camera side by side. The main camera captures a focused image, while the secondary camera captures a defocused image.

[0008] Step 2: Use the RAFT optical flow method to align the focused image from the main camera and the out-of-focus image from the secondary camera to resolve differences in displacement and occlusion between the two images; specifically, use the calculated optical flow to align the I... F To I D Alignment, then based on the occlusion area identifier map M, using I D Pixel information filling I F The occluded area is used to obtain a focused image I after alignment and correction of the occluded area. A ;

[0009] Step 3: Place I A and I D Input the demoiré network, and I D As a guide, let the network learn to remove moiré patterns. R ;

[0010] Step 4: Recovery based on joint bilateral filtering, I R As a guide, for I A Perform joint bilateral filtering to generate the final output image I O Since filtering is in I A Executed on, therefore I O In terms of brightness and color, it is similar to I A It is consistent, and since the filter weights are based on I R This setting ensures the removal of moiré patterns.

[0011] As described above, the image demoiring method based on a dual-camera system with both focus and defocus features no specific parameter requirements for the main camera, while the secondary camera needs to have a large aperture to produce a shallow depth of field and focus on areas outside the LCD / OLED screen in the scene.

[0012] As described above, in the image demoiring method based on a dual-camera system with both focus and defocus capabilities, step two involves I... DAlthough the filled occluded area is out of focus, it is usually very small because the parallax between the main and secondary cameras is very small. Moreover, the occluded area is usually not in the foreground area that is the focus of visual attention, but at the boundary between the background and the foreground. Therefore, its subjective visual impact is negligible.

[0013] As described above, the image demoiring method based on a dual-camera system with both focus and defocus features a multi-dimensional loss function in step three of the network learning process. This multi-dimensional loss function enhances the effectiveness of demoiring and includes consistency loss, perceptual loss, high-frequency loss, and adversarial loss.

[0014] The image demoirée method based on a dual-camera system with both focus and defocus, as described above, employs an L1 loss mechanism to ensure consistency between the output and the actual data. The specific function of L1 loss is: C =||I R -GT||1, where I R The final output of the network is GT, which represents the training and labeled images.

[0015] As described above, the image demoiring method based on a dual-camera system with both focused and unfocused focus utilizes a pre-trained VGG16 network Φ to extract features and design a multi-scale loss function. The specific function of this loss function is as follows: in, It is the output of the i-th layer, GT i These are the corresponding labeled images. Utilizing the prior information from VGG can improve the subjective effect of the results, especially in regions with obvious semantic features.

[0016] As described above, the image demoiring method based on a dual-camera system with both focus and defocus capabilities uses a high-frequency loss function, L, to better correct high-frequency information in the result. H =||FFT(I R )-FFT(GT)||1.

[0017] The image demoirée method based on a dual-camera system with both focus and defocus, as described above, incorporates an adversarial loss that integrates multi-scale input and multi-scale output adversarial losses. Its specific function is as follows: Where i is the level of the input scale, j is the level of the output scale, L is the total number of levels, and D ij V is the corresponding estimation result obtained by the discriminator at input scale i and output scale j. ij It is to input the corresponding "true" or "false" label.

[0018] The image demoirée method based on a dual-camera system with focused and unfocused focus, as described above, uses a multidimensional loss function L.total =L C +λ P L P +λ H L H +λ A L A , where λ P , λ H and λ A It is a hyperparameter that controls the relative weights.

[0019] The image demoirée method based on a dual-camera system with focused and unfocused focus, as described above, wherein λ P The value is set to 1, and the λ mentioned above H The set value is 0.1, and the λ mentioned above. A The value is set to 0.1.

[0020] The advantages of this invention are: This invention uses a dual-camera system to acquire focused and out-of-focus images. The focused image contains high-quality texture but may have moiré patterns, while the out-of-focus image has significantly reduced moiré patterns but the texture is somewhat blurred. By performing demoiré processing on the focused and out-of-focus images, the moiré patterns of the images can be removed accurately and efficiently. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a diagram illustrating moiré patterns and real textures;

[0023] Figure 2 This is a schematic diagram of the device and focusing / defocusing of an embodiment of the present invention (where the left figure is a schematic diagram of the device; the right figure is a schematic diagram of the focused and defocused images; Figures (a) and (b) are used to illustrate the problems of brightness / color inconsistency and occlusion between the focused and defocused images, respectively).

[0024] Figure 3 This is a schematic diagram of the moiré-removing frame according to an embodiment of the present invention;

[0025] Figure 4 Figure 1 is a schematic diagram of the network structure of an embodiment of the present invention (where Figure (a) is a schematic diagram of the demoiralized network; Figure (b) is a schematic diagram of the discriminator in the adversarial loss):

[0026] Figure 5 This is a schematic diagram illustrating the imaging principle of an embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram illustrating the processing and generation of data using the method of this invention in an embodiment of the invention;

[0028] Figure 7 This is one of the schematic diagrams showing the results of different methods on synthetic data in the verification test of this invention (the area marked with a green box in each image is magnified);

[0029] Figure 8 This is the second schematic diagram of the results of different methods on the synthesized data in the verification test of the present invention (the area marked with a green box in each image is enlarged);

[0030] Figure 9 This is the third schematic diagram of the results of different methods on the synthesized data in the verification test of the present invention (the area marked with a green box in each image is enlarged);

[0031] Figure 10 This is the fourth illustration of the results of different methods on the synthesized data in the verification test of the present invention (the area marked with a green box in each image is magnified). Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Example

[0034] Real data acquisition device

[0035] like Figure 2 As shown on the left, the dual-camera system designed in this invention, consisting of two cameras placed side-by-side, comprises a main camera capturing a focused image and a secondary camera capturing a defocused image. This system offers users flexibility as it does not impose specific requirements on the main camera, allowing users to freely choose their camera and imaging parameters. However, it is important to note that some limitations need to be imposed on the secondary camera to capture defocused images with significantly weaker moiré patterns. It requires a large aperture to produce a shallow depth of field and to focus on areas outside the LCD / OLED screen in the scene. Figure 2 (Right) provides examples of images in true focus and out of focus. By combining the examples, it can be seen that there are inconsistencies in brightness / color and occlusion issues between the two images, which is also the problem that this invention needs to solve.

[0036] Moiré frame

[0037] The moiré-removing frame proposed in this invention is as follows: Figure 3 As shown, the entire process utilizes the contextual information between the two images and completes the process in three steps: alignment, demoiring, and restoration.

[0038] The first step is image alignment. This invention uses the RAFT optical flow method to align images and resolve differences such as displacement and occlusion between two images. (It should be noted that because the calibration of binocular cameras is time-consuming and easily affected by aberrations and relative movement between the primary and secondary cameras, this invention does not use a stereo matching algorithm.) In the first alignment step, after calculating the optical flow, a natural alignment method is to use the out-of-focus image I captured by the secondary camera... D Focused image I captured by the main camera F Alignment is an option, but this method has significant limitations. Due to occlusion areas caused by parallax, the out-of-focus image lacks corresponding pixel information to guide the processing of pixels in these occluded areas. Therefore, processing of occluded areas often degenerates into a single image demoiring problem, which contradicts the original intent of this invention. Therefore, this invention chooses the opposite alignment method, namely, using calculated optical flow to align I... F To I D Alignment, then based on the occlusion area identification map M (calculated through standard left and right checks), use I... D Pixel information filling I F The occluded area is used to obtain a focused image I after alignment and correction of the occluded area. A Here, I D Although the filled occluded area is out of focus, it is usually very small because the parallax between the main and secondary cameras is very small. Moreover, the occluded area is usually not in the foreground area that is the focus of visual attention, but at the boundary between the background and the foreground. Therefore, its subjective visual impact is negligible.

[0039] The second step is to remove moiré patterns. Based on the first step, this invention will... A and I D Input the demoiré network, and I D As a guide, let the network learn to remove moiré patterns. R In addition, this invention also designs a multidimensional loss function to enhance the effectiveness of demoiring, including consistency loss, perceptual loss, high-frequency loss and adversarial loss.

[0040] Demoiré network model

[0041] The demoiré network model designed in this invention is as follows: Figure 4 (a) The entire network structure is based on the UNet architecture. To enhance network performance, expanded residual dense blocks and scale-aware blocks (SAM) are used in each processing layer. The multidimensional loss used during model training is described below:

[0042] Consistency Loss: To ensure consistency between the output and the actual data, this invention employs L1 loss, i.e.: L C =||I R -GT||1, where I R The final output of the network is GT, which represents the training and labeled images.

[0043] Perceptual Loss: To improve the perceptual quality of the results, this invention is inspired by the use of a pre-trained VGG16 network Φ to extract features and design a multi-scale loss, namely: in, It is the output of the i-th layer, GT i These are the corresponding labeled images. Utilizing the prior information from VGG can improve the subjective effect of the results, especially in regions with obvious semantic features (such as sky and water).

[0044] High-frequency loss: To better correct for high-frequency information in the results, high-frequency loss is used, i.e., L H =||FFT(I R )-FFT(GT)||1.

[0045] Adversarial Loss: In addition to the manually designed loss mentioned above, this invention also designs a discriminator to automatically learn whether the result contains moiré patterns. To train this discriminator, this invention uses a defocused image I D And the results of moiré removal I R As input, and labeled as "false", the defocused image I D The trained labeled image GT is used as input and labeled as "true". The discriminator's network structure is as follows: Figure 4 (b) Inspired by GigaGAN, this invention integrates multi-scale input and multi-scale output adversarial loss, namely: Where i is the level of the input scale, j is the level of the output scale, L is the total number of levels, and D ij V is the corresponding estimation result obtained by the discriminator at input scale i and output scale j. ij It is to input the corresponding "true" or "false" label.

[0046] The final loss function can be expressed as the sum of four components: L total =L C +λ P L P +λH L H +λ A L A , where λ P , λ H and λ A These are hyperparameters that control the relative weights. In this invention, λ is set respectively. P , λ H and λ A The values ​​are 1, 0.1, and 0.1.

[0047] The third and final step is the recovery process. This step is designed because the input during training is the focused image I. A and defocused image I D Output image I R Possibly affected by inconsistencies in brightness and color between the two, compared to I A Differences arise in brightness and color. This invention designs a recovery process based on joint bilateral filtering, which... R As a guide, for I A Joint bilateral filtering

[34] is performed to generate the final output image I O Since filtering is in I A Executed on, therefore I O In terms of brightness and color, it is similar to I A It is consistent, and since the filter weights are based on I R This setting ensures the removal of moiré patterns.

[0048] Generation of synthetic data

[0049] Since real-world data lacks corresponding images without moiré patterns, making them unsuitable for model training, this invention designs a method for synthesizing moiré pattern data and creates a synthetic dataset for model training. Figure 5 This demonstrates the generation process of a focused image (clear texture but heavy moiré) and a defocused image (blurred texture but weak moiré), where, for the original image signal... Figure 5 (a) The signal emitted by the LCD / OLED screen is as follows Figure 5 As shown in (b), since the red (R), green (G), and blue (B) emission units are arranged side-by-side in space, the light signal emitted by the green emission unit from the repeating LCD / OLED emission units forms a periodically changing light signal that spatially spans the screen in the G channel. During the generation of the focused image, signal (b) lies within the camera's depth of field, allowing light emitted from any position to propagate to the same position on the focal plane, thus enabling clear imaging, such as... Figure 5As shown in (c). However, during the generation of the out-of-focus image, signal (b) is outside the camera's depth of field. According to the point spread function (PSF), the emitted light rays will not propagate to the spatial position corresponding to the emission position, resulting in out-of-focus blur. This out-of-focus blur not only blurs the texture of signal (a) but also weakens the obvious spatial variation of the light signal caused by the spatial juxtaposition of the RGB light-emitting units on the screen. This makes the light intensity of adjacent positions tend to be smoother, such as... Figure 5 As shown in (c). When the optical signal is subsequently discretely sampled by CCD / CMOS on the focal plane, if the sampling frequency is similar to the repetition frequency of the optical signal, the discretely sampled signal will exhibit obvious moiré patterns in the focused image, such as... Figure 5 As shown in (d). In a defocused image, due to the defocusing and blurring, the repetitive changes of the blurred optical signal are greatly reduced or even disappear. Therefore, the probability of the discrete sampled signal appearing in the defocused image is greatly reduced.

[0050] To simulate light signals emitted from an LCD / OLED screen, this invention converts the original image into an RGB subpixel arrangement and performs a controlled random projection transformation on it. To simulate imaging from a data camera, radial distortion is used to simulate lens distortion. Simultaneously, when generating the out-of-focus image, a Gaussian blur with a 9×9 kernel and a standard deviation of 1.6 is used to simulate defocusing. Next, a Bayer color filter array is used to sample the image, and the sampled image is decoded. Finally, JPEG compression noise is added to the decoded image (with a quantization factor of 0.5 and a pixel block size of 8×8) to obtain the final focused and out-of-focus images, which are used to construct a synthetic dataset for training.

[0051] Based on the above moiré pattern generation method, this invention generated 4627 sets of images. However, subsequent research revealed that most of the generated moiré patterns were colored, meaning that the colors of adjacent pixels differed significantly. This could lead to overfitting the model during training to colored moiré patterns, which is problematic because black and white moiré patterns may also exist in real images. To address this issue, this invention additionally generated focused and out-of-focus images with black and white moiré patterns. Specifically, the colored moiré pattern image generated according to the above method and the original input (excluding moiré patterns) were converted to the YCbCr space, and the CbCr channel of the moiré pattern image was replaced with the CbCr channel of the original input, thus obtaining focused and out-of-focus images with black and white moiré patterns. A total of 3059 sets of such data were obtained. Furthermore, foreground elements such as presenters and actors may actually appear on LCD / OLED screens. Demoiré models might blur foreground details. To improve the model's ability to retain foreground details, this invention uses graphics software to generate foreground cartoon images and places them in both focused and out-of-focus images. The foreground cartoon image also becomes correspondingly blurred in the out-of-focus image. The entire synthetic dataset, after supplementation, contains 7686 images; this work places the foreground in 2325 of these images. Figure 6 Some examples are given.

[0052] Verification test

[0053] The algorithm designed in this invention is implemented based on PyTorch, and all experiments were conducted on an NVIDIA RTX 3090 GPU. The model uses Adam as the optimizer. During training, images were cropped to 256×256 pixels, with a batch size of 1, 100 iterations, and an initial learning rate of 0.0002. This invention designs three evaluation metrics: PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity), and LPIPS (Learning-Aware Patch Similarity) to objectively evaluate the effectiveness of this invention.

[0054] To demonstrate the effectiveness of this invention, it is compared with the latest and most classic related works on the created synthetic dataset and real data. Related works include single-image demoiring algorithms DMCNN, MopNet, MBCNN, WDNet, ESDNet, and FDNet; multi-image demoiring algorithm MMDM; video demoiring algorithm VDMPCD; and general single-image image restoration algorithms Uformer, SiamTrans, and SwinIR, and general multi-image restoration methods BPN and MFD. All methods were retrained on the synthetic dataset. For single-image algorithms, the input was a focused image; for multi-image algorithms, the input was a focused image and a defocused image. The PSNR (higher values ​​indicate better performance), SSIM (higher values ​​indicate better performance), and LPIPS (lower values ​​indicate better performance) of the above methods were statistically analyzed, and the results are shown in Table 1.

[0055] DMCNN MopNet MBCNN WDNet Uformer ESDNet SwinIR PSNR 28.56 29.47 24.80 26.54 29.56 29.74 28.15 SSIM 0.8166 0.8538 0.6227 0.6624 0.8336 0.8485 0.7971 LPIPS 0.2482 0.1705 0.3963 0.2128 0.2022 0.1579 0.2785 MFD BPN MMDM SiamTrans VDMPCD FDNet This invention PSNR 28.47 31.31 22.38 23.78 32.38 26.44 34.38 SSIM 0.8277 0.8746 0.4839 0.6871 0.8708 0.6717 0.9151 LPIPS 0.2431 0.1480 0.4943 0.3842 0.1510 0.3481 0.1243

[0056] Table 1

[0057] As shown in Table 1 and Figure 7-10 As shown, single-image demoiring algorithms and single-image image restoration algorithms perform poorly. This is because in real-world scenes, single-image algorithms often struggle to distinguish between moiré patterns and real textures. Since out-of-focus images provide significantly less image pixel information for moiré patterns, multi-image demoiring algorithms (MMDM) and multi-image restoration methods (MFD) also perform poorly. Although multi-image restoration methods (BPN) and video demoiring algorithms (VDMPCD) objectively outperform other single-layer and multi-layer algorithms, they still differ somewhat from the data presented in this invention. Furthermore, through… Figure 7-10 It is evident that the moiré pattern removal effect of the final demoirée image obtained by this invention is significantly superior to that of other single-image and multi-image algorithms. Therefore, the multi-image algorithm designed in this invention achieves optimal results in terms of network, loss function, and framework design, and outperforms other multi-image and single-image algorithms, thereby realizing accurate removal of moiré patterns from images.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for removing moiré patterns from images based on a dual-camera system with both focused and defocused focus, characterized in that: Includes the following steps: Step 1: Set up the main camera and the secondary camera side by side. The main camera captures a focused image, while the secondary camera captures a defocused image. Step 2: Align the focused image captured by the main camera and the out-of-focus image captured by the secondary camera using the optical flow method RAFT. Specifically, the calculated optical flow is used to align the focused image... To the out-of-focus image Alignment, then based on the occlusion area marker M, using... Pixel information filling The occluded area is used to obtain a focused image after alignment and correction of the occluded area. The obstructed area is created by the parallax between the main and secondary cameras and is located at the boundary between the background and the foreground. Step 3: Put and Input the demoiré network, and As a guide, the demoiring results are obtained through a demoiring network. ; The training of the demoiroid network in step three includes a multidimensional loss function, which includes consistency loss, perceptual loss, high-frequency loss and adversarial loss. The consistency loss adopts L1 loss, that is: ,in, For the final output of the network, For training labeled images; The perceived loss: using a VGG16 network Extract features and design a multi-scale loss, namely: ,in, It is the output of the i-th layer of the demoirtication network. These are the corresponding labeled images; The high-frequency loss refers to the high-frequency information in the correction result, which is then processed using high-frequency loss, i.e.: Where FFT is the two-dimensional Fast Fourier Transform; The adversarial loss: to remove the out-of-focus image and moiré pattern removal results As input to the discriminator and labeled as "false", the out-of-focus image is... and training labeled images As input to the discriminator and marked as "true", it combines multi-scale input and multi-scale output adversarial loss, i.e.: Where i is the level of the input scale, j is the level of the output scale, and L is the total number of levels. These are the corresponding estimation results obtained by the discriminator at input scale i and output scale j. These are the corresponding "true" or "false" labels; The final loss function can be expressed as the sum of four components: ,in, , and It is a hyperparameter that controls the relative weights; Step 4: Recovery based on joint bilateral filtering, As a guide, Perform joint bilateral filtering to generate the final output image. , In terms of brightness and color Consistent.

2. The image demoiring method based on a dual-camera system with focused and unfocused focus as described in claim 1, characterized in that: The main camera has no specific parameter requirements, but the secondary camera needs to have a large aperture to produce a shallow depth of field and focus on areas outside the LCD / OLED screen in the scene.

3. The image demoiring method based on a dual-camera system with focused and unfocused modes according to claim 1, characterized in that: In step two as described The filled, occluded area is out of focus.

4. The image demoiring method based on a dual-camera system with focused and unfocused focus as described in claim 1, characterized in that: The aforementioned The set value is 1, the aforementioned The set value is 0.1, as stated above. The value is set to 0.1.

Images