An integrated imaging light field denoising method based on cross-region joint unsupervised learning

Abstract

This invention proposes an integrated imaging light field denoising method based on cross-regional joint unsupervised learning. The method comprises five processes: acquisition of a noisy micro-image array, separation and cross-regional pairing of noisy micro-images, feature block selection and training pair construction, residual Transformer unsupervised light field denoising, and 3D reconstruction of the denoised micro-image array. The method acquires a noisy micro-image array as input, separates the micro-images based on the spatial arrangement of the lens array, establishes cross-regional candidate pairing relationships, constructs cross-regional training pairs by combining effective region masking, candidate feature block extraction, similarity calculation, and geometric consistency filtering, performs unsupervised light field denoising using a residual Transformer denoising network, and outputs a denoised micro-image array. Finally, the corresponding 3D scene is reconstructed using an integrated imaging display device, improving the quality of integrated imaging light field denoising and 3D reconstruction without requiring a clean ground truth image.

Description

Technical Field

[0001] This invention relates to the field of image processing for computer vision tasks, and more specifically, to an integrated imaging light field denoising method using cross-regional joint unsupervised learning. Background Technology

[0002] Integrated imaging, as a crucial method for realizing 3D light field displays, utilizes lens arrays to acquire spatial and angular information of 3D scenes and encodes it into a 2D micro-image array. During the reconstruction phase, the principle of optical path reversibility is used to restore the 3D display result of the target scene. Due to its ability to provide rich depth cues and multi-view information, integrated imaging has shown broad application prospects in 3D display, microscopic imaging, target detection, and augmented reality. However, in actual integrated imaging acquisition, the micro-image array is inevitably contaminated by various noise sources, including image sensor readout noise, photon shot noise, ambient lighting fluctuations, and inherent noise of the optical system. This noise not only degrades the imaging quality of individual micro-images but also disrupts the parallax consistency that should be maintained between different micro-images. This causes random fluctuations in the observation results of the same spatial point under different viewpoints, leading to artifacts, edge blurring, and loss of texture details in subsequent 3D reconstruction, severely impacting the display quality and practical application effects of the integrated imaging system.

[0003] Existing image denoising methods mainly include traditional filtering methods, supervised learning methods, and unsupervised learning methods. Traditional methods have limited adaptability to complex noise and multi-view geometric relationships; supervised learning methods usually rely on a large number of precisely paired clean and noisy images, but such data is difficult to obtain in real integrated imaging systems; although existing unsupervised methods have to some extent gotten rid of the dependence on clean labels, most of them only use the similarity between local adjacent views to construct training pairs, which makes it difficult to fully explore the long-range structural correlations in the global scope of micro-image arrays. Therefore, in complex scenes and under strong noise conditions, they are still prone to problems such as over-smoothing, loss of detail, and insufficient preservation of geometric consistency. Summary of the Invention

[0004] This invention proposes an integrated imaging light field denoising method based on cross-regional joint unsupervised learning. The core of this method lies in constructing cross-regional training pairs with consistent content and independent noise through cross-regional pairing, and then using a residual Transformer denoising network to perform unsupervised light field denoising on a noisy micro-image array. The method first obtains the noisy micro-image array as input. Then, it performs noisy micro-image separation and cross-regional pairing on the noisy micro-image array to obtain a cross-regional candidate block set. Next, it performs effective region mask generation, candidate feature block extraction, similarity calculation, and geometric consistency filtering on the cross-regional candidate blocks to construct cross-regional training pairs. Then, it trains the residual Transformer denoising network using the cross-regional training pairs as input, and uses the trained residual Transformer denoising network to process the noisy micro-image array, outputting a denoised micro-image array. Finally, it loads the denoised micro-image array into an integrated imaging display device to reconstruct the corresponding 3D scene.

[0005] The method comprises five processes: acquisition of noisy micro-image arrays, separation and cross-region pairing of noisy micro-images, feature block selection and training pair construction, residual Transformer unsupervised light field denoising, and 3D reconstruction of the denoised micro-image array. The specific process is attached. Figure 1 As shown.

[0006] The noisy micro-image array acquisition process involves acquiring the light field of the target 3D scene using a real integrated imaging acquisition system to obtain a noisy micro-image array. Under virtual scene conditions, a virtual camera array is established using computer graphics software to generate multi-view images. Preset noise is superimposed on these multi-view images, and then the images are reconstructed according to the spatial angular correspondence of the lens array to obtain the noisy micro-image array. This noisy micro-image array is then used as input for subsequent denoising processing.

[0007] The noisy micro-image separation and cross-region pairing process is as follows: Figure 2 As shown in the diagram, the process first separates the noisy micro-image array based on the spatial arrangement of the lens array, resulting in a sequence of micro-image units composed of multiple micro-image units. Then, using the micro-image units as the basic search objects, a cross-regional candidate block with similar structural content to the current micro-image unit region is searched within the micro-image unit sequence, and candidate pairing relationships are established to obtain a set of cross-regional candidate blocks.

[0008] The feature block selection and training process is described in the attached figure. Figure 3As shown, this process generates effective region masks for cross-region candidate blocks and obtains candidate feature blocks by preserving edge, contour, and texture regions based on the effective region masks. Subsequently, feature extraction and similarity calculation are performed on the candidate feature blocks. Furthermore, based on the preliminary screening results, geometric consistency constraints are combined to filter candidate feature block pairs, retaining candidate feature block pairs with consistent content and independent noise, thus constructing cross-region training pairs.

[0009] The residual Transformer unsupervised optical field denoising process is as follows: Figure 4 As shown in the figure. This process constructs a cross-region training input residual Transformer denoising network. Specifically, the residual Transformer denoising network adopts a multi-scale structure of encoder-bottleneck layer-decoder. In the encoding stage, image features of different scales are extracted step by step through downsampling. In the bottleneck layer, cross-region image patch features are aggregated. In the decoding stage, spatial resolution is gradually restored through upsampling and the denoising result is output. Further, the core modules of the residual Transformer denoising network include a shifted window multi-head self-attention unit and a gated convolutional feedforward unit, as shown in the figure. Figure 5 As shown, the shifted-window multi-head self-attention unit divides the input features into multiple local windows, calculates multi-head self-attention within each local window, and shifts the window positions between adjacent network layers, allowing features within different local windows to interact. The gated convolutional feedforward unit sequentially performs convolutional mapping, gated activation, and convolutional fusion on the self-attention output features to obtain the fused features. The residual Transformer denoising network takes a noisy micro-image array as input, outputs a corresponding noise residual map, and performs residual operations between the noise residual map and the input noisy micro-image array to obtain a denoised micro-image array.

[0010] The 3D reconstruction process of the denoised micro-image array involves loading the denoised micro-image array obtained through the above processing onto an integrated imaging display device. The integrated imaging display device performs optical reconstruction according to the correspondence between each micro-image unit in the denoised micro-image array and the lens array, reconstructing the denoised micro-image array into a corresponding 3D scene.

[0011] This invention proposes an integrated imaging light field denoising method based on cross-regional joint unsupervised learning. The method completes the processing flow from noisy micro-image array input to 3D scene reconstruction through noisy micro-image array acquisition, noisy micro-image separation and cross-regional pairing, feature block selection and training pair construction, residual Transformer unsupervised light field denoising, and 3D reconstruction of the denoised micro-image array. This method achieves high-quality light field recovery without requiring clean ground truth images, improving denoising capabilities under complex noise conditions while also preserving texture details and maintaining multi-view structural consistency, providing an effective technical solution for high-fidelity 3D display in integrated imaging systems. Attached Figure Description

[0012] Appendix Figure 1 This is a flowchart of an integrated imaging light field denoising method based on cross-regional joint unsupervised learning proposed in this invention.

[0013] Appendix Figure 2 This is a flowchart of the process of separating noisy micro-images and pairing across regions.

[0014] Appendix Figure 3 Construct a framework diagram for feature block selection and training.

[0015] Appendix Figure 4 This is a schematic diagram of the residual Transformer denoising network structure.

[0016] Appendix Figure 5 This is a schematic diagram of the core module of the residual Transformer denoising network.

[0017] It should be understood that the above figures are only schematic and are not drawn to scale. Detailed Implementation

[0018] The following detailed description of a typical embodiment of the integrated imaging optical field denoising method based on cross-regional joint unsupervised learning proposed in this invention further illustrates the invention. It is important to note that the following embodiments are for illustrative purposes only and should not be construed as limiting the scope of protection of this invention. Any non-essential improvements and adjustments made to this invention by those skilled in the art based on the above description are still within the scope of protection of this invention.

[0019] This invention proposes an integrated imaging light field denoising method based on cross-regional joint unsupervised learning. The core of this method lies in constructing cross-regional training pairs with consistent content and independent noise through cross-regional pairing, and then using a residual Transformer denoising network to perform unsupervised light field denoising on a noisy micro-image array. The method first obtains the noisy micro-image array as input. Then, it performs noisy micro-image separation and cross-regional pairing on the noisy micro-image array to obtain a cross-regional candidate block set. Next, it performs effective region mask generation, candidate feature block extraction, similarity calculation, and geometric consistency filtering on the cross-regional candidate blocks to construct cross-regional training pairs. Then, it trains the residual Transformer denoising network using the cross-regional training pairs as input, and uses the trained residual Transformer denoising network to process the noisy micro-image array, outputting a denoised micro-image array. Finally, it loads the denoised micro-image array into an integrated imaging display device to reconstruct the corresponding 3D scene.

[0020] The method comprises five processes: acquisition of noisy micro-image arrays, separation and cross-region pairing of noisy micro-images, feature block selection and training pair construction, residual Transformer unsupervised light field denoising, and 3D reconstruction of the denoised micro-image array. The specific process is attached. Figure 1 As shown.

[0021] The noisy micro-image array acquisition process involves acquiring the light field of the target 3D scene using a real integrated imaging acquisition system to obtain a noisy micro-image array. Under virtual scene conditions, a virtual camera array is established using computer graphics software to generate multi-view images. Preset noise is superimposed on these multi-view images, and then the images are reconstructed according to the spatial angular correspondence of the lens array to obtain the noisy micro-image array. This noisy micro-image array is then used as input for subsequent denoising processing.

[0022] The noisy micro-image separation and cross-region pairing process is as follows: Figure 2 As shown. First, the noisy micro-image array is separated according to the spatial arrangement of the lens array, resulting in a sequence of micro-image units composed of multiple micro-image units. In this embodiment, the size of a single micro-image unit is set to 32×32 pixels. Adjacent 4×4 micro-image units are combined into a cross-regional candidate block, and a sliding search is performed in the corresponding micro-image unit sequence of the entire micro-image array with a step size of 2 micro-image units. Subsequently, based on the current cross-regional candidate block, cross-regional candidate blocks with similar structural content are searched within the micro-image unit sequence, and candidate pairing relationships are established to obtain a set of cross-regional candidate blocks.

[0023] The feature block selection and training process is described in the attached figure. Figure 3As shown, this process generates an effective region mask for cross-region candidate blocks and preserves edge, contour, and texture regions based on the effective region mask to obtain candidate feature blocks. In this embodiment, a two-dimensional discrete cosine transform is performed on each candidate feature block, and the 8×8 low-frequency coefficients in the upper left corner are truncated as structural features. The similarity between candidate feature blocks is calculated based on the structural features. For each candidate feature block, the top 5 candidate feature blocks with the highest similarity are selected as the initial matching set. Further, based on the initial matching set, a geometric consistency constraint is used for filtering to keep the relative offset between candidate feature block pairs within one-quarter of the candidate block size. Finally, candidate feature block pairs with consistent content and independent noise are retained to construct cross-region training pairs.

[0024] The residual Transformer unsupervised optical field denoising process is as follows: Figure 4 As shown, this process constructs a cross-region training input residual Transformer denoising network. Specifically, the residual Transformer denoising network adopts a multi-scale structure of encoder-bottleneck layer-decoder. In the encoding stage, image features of different scales are extracted step by step through downsampling. In the bottleneck layer, cross-region image patch features are aggregated. In the decoding stage, spatial resolution is gradually restored through upsampling, and the denoising result is output.

[0025] Furthermore, the core modules of the residual Transformer denoising network include a shifted window multi-head self-attention unit and a gated convolutional feedforward unit, as shown in the attached figure. Figure 5 As shown, the shifted-window multi-head self-attention unit divides the input features into multiple local windows, calculates multi-head self-attention within each local window, and shifts the window positions between adjacent network layers, allowing features within different local windows to interact. The gated convolutional feedforward unit sequentially performs convolutional mapping, gated activation, and convolutional fusion on the self-attention output features to obtain the fused features. The residual Transformer denoising network takes a noisy micro-image array as input, outputs a corresponding noise residual map, and performs residual operations between the noise residual map and the input noisy micro-image array to obtain a denoised micro-image array.

[0026] During network training, one noisy image patch from the cross-region training pair is used as input, and another noisy image patch with identical content but independent noise is used as the supervised target to perform unsupervised training on the residual Transformer denoising network. After training, the complete noisy micro-image array is input into the trained residual Transformer denoising network according to the aforementioned cross-region candidate block scale for inference to obtain the corresponding denoising candidate blocks. For overlapping regions between adjacent denoising candidate blocks, they are fused according to the overlapping pixel positions to obtain the complete denoised micro-image array.

[0027] The 3D reconstruction process of the denoised micro-image array involves loading the denoised micro-image array obtained through the above processing onto an integrated imaging display device. The integrated imaging display device performs optical reconstruction according to the correspondence between each micro-image unit in the denoised micro-image array and the lens array, reconstructing the denoised micro-image array into a corresponding 3D scene.

[0028] This invention proposes an integrated imaging light field denoising method based on cross-regional joint unsupervised learning. The method completes the processing flow from noisy micro-image array input to 3D scene reconstruction through noisy micro-image array acquisition, noisy micro-image separation and cross-regional pairing, feature block selection and training pair construction, residual Transformer unsupervised light field denoising, and 3D reconstruction of the denoised micro-image array. This method achieves high-quality light field recovery without requiring clean ground truth images, improving denoising capabilities under complex noise conditions while also preserving texture details and maintaining multi-view structural consistency, providing an effective technical solution for high-fidelity 3D display in integrated imaging systems.

Claims

1. An integrated imaging light field denoising method based on cross-regional joint unsupervised learning, characterized in that, The method includes five processes: acquisition of noisy micro-image arrays, separation and cross-region pairing of noisy micro-images, feature block screening and training pair construction, residual Transformer unsupervised light field denoising, and 3D reconstruction of the denoised micro-image array. In the acquisition of noisy micro-image arrays, noisy micro-image arrays are acquired as input. In the separation and cross-region pairing of noisy micro-images, the noisy micro-image arrays are separated to obtain a sequence of micro-image units, and cross-region candidate blocks are searched within the range of the micro-image unit sequence to establish candidate pairing relationships and obtain a set of cross-region candidate blocks. In the feature block selection and training pair construction process, effective region mask generation, candidate feature block extraction, similarity calculation, and geometric consistency filtering are performed on the cross-region candidate blocks to construct cross-region training pairs. In the residual Transformer unsupervised light field denoising process, the cross-region training pairs are input to the residual Transformer denoising network for training, and the trained residual Transformer denoising network is used to process the noisy micro-image array to output a denoised micro-image array. In the 3D reconstruction process of the denoised micro-image array, the denoised micro-image array is loaded onto the integrated imaging display device to reconstruct the denoised micro-image array into the corresponding 3D scene.

2. The integrated imaging light field denoising method based on cross-regional joint unsupervised learning according to claim 1, characterized in that, In the process of acquiring noisy micro-image arrays, the light field of the target three-dimensional scene is acquired by a real integrated imaging acquisition system to obtain noisy micro-image arrays; In a virtual scene, a virtual camera array is established using computer graphics software to generate multi-view images. Preset noise is then superimposed on the multi-view images, and the multi-view images are recombined according to the spatial angle correspondence of the lens array to obtain a noisy micro-image array.

3. The integrated imaging light field denoising method based on cross-regional joint unsupervised learning according to claim 1, characterized in that, In the process of noisy micro-image separation and cross-region pairing, the noisy micro-image array is separated according to the spatial arrangement relationship of the lens array to obtain a sequence of micro-image units composed of multiple micro-image units. Then, taking the micro-image units as the basic search objects, cross-region candidate blocks with similar structural content to the current micro-image unit region are searched within the range of the micro-image unit sequence, and candidate pairing relationships are established to obtain a set of cross-region candidate blocks.

4. The integrated imaging light field denoising method based on cross-regional joint unsupervised learning according to claim 1, characterized in that, In the process of feature block screening and training pair construction, an effective region mask is generated for cross-region candidate blocks, and the edge, contour and texture regions are preserved according to the effective region mask to obtain candidate feature blocks; Subsequently, feature extraction and similarity calculation are performed on the candidate feature blocks. Furthermore, based on the preliminary screening results, the candidate feature block pairs are filtered by combining geometric consistency constraints, retaining candidate feature block pairs with consistent content and independent noise, and constructing cross-region training pairs.

5. The integrated imaging light field denoising method based on cross-regional joint unsupervised learning according to claim 1, characterized in that, In the unsupervised light field denoising process of residual Transformer, the constructed cross-region training pair is input into the residual Transformer denoising network. The residual Transformer denoising network adopts a multi-scale structure of encoder-bottleneck layer-decoder. In the encoding stage, image features of different scales are extracted step by step through downsampling. In the bottleneck layer, cross-region image patch features are aggregated. In the decoding stage, spatial resolution is gradually restored through upsampling and the denoising result is output.

6. The integrated imaging light field denoising method based on cross-regional joint unsupervised learning according to claim 1, characterized in that, The core modules of the residual Transformer denoising network include a shift-window multi-head self-attention unit and a gated convolutional feedforward unit. The shift-window multi-head self-attention unit divides the input features into multiple local windows, calculates multi-head self-attention within each local window, and shifts the window positions between adjacent network layers to allow features within different local windows to interact. The gated convolutional feedforward unit sequentially performs convolution mapping, gated activation, and convolutional fusion on the self-attention output features to obtain fused features. The residual Transformer denoising network takes a noisy micro-image array as input, outputs a corresponding noise residual map, and performs residual operations between the noise residual map and the input noisy micro-image array to obtain a denoised micro-image array.

7. The integrated imaging light field denoising method based on cross-regional joint unsupervised learning according to claim 1, characterized in that, In the 3D reconstruction process of the denoised micro-image array, the denoised micro-image array obtained by the residual Transformer unsupervised light field denoising process is loaded onto the integrated imaging display device; the integrated imaging display device performs optical reconstruction according to the correspondence between each micro-image unit in the denoised micro-image array and the lens array, and reconstructs the denoised micro-image array into the corresponding 3D scene.

Images