A light field image denoising method and device based on a double-branch hybrid architecture

Abstract

The present disclosure relates to the technical field of computer vision and image processing, and provides a light field image denoising method and device based on a double-branch hybrid architecture, which comprises the following steps: preprocessing an original light field image to generate an original-noise image pair, and taking the original-noise image pair as a training data set; constructing a light field image denoising network model of a double-branch hybrid architecture, wherein the network model comprises a local modeling branch, a global modeling branch, a feature fusion module and a progressive multi-scale reconstruction module; training the light field image denoising network model according to the training data set by using a progressive training strategy to obtain a trained light field image denoising network model; and inputting a noisy light field image to be processed into the trained light field image denoising network model to output a reconstructed denoised image, so that efficient denoising is realized while the integrity of the light field geometric structure is maintained.

Description

Technical Field

[0001] This disclosure relates to the fields of computer vision and image processing technology, and in particular to a method and apparatus for light field image denoising based on a dual-branch hybrid architecture. Background Technology

[0002] Light field images, as a novel image format capable of simultaneously recording spatial location and light direction information, have significant application value in fields such as computer vision, virtual reality, and 3D reconstruction. However, during acquisition and transmission, light field images are susceptible to sensor noise, low-light environments, and transmission interference, leading to the introduction of various types of noise into the images. The presence of noise not only degrades the visual quality of light field images but also severely interferes with subsequent advanced visual tasks such as multi-view matching and depth estimation, thereby reducing the application value of light field data.

[0003] Existing light field image denoising methods can be mainly divided into traditional methods and deep learning-based methods. Traditional methods, such as filtering algorithms based on sparse representation and non-local similarity, can suppress noise to a certain extent, but they are difficult to effectively model the complex spatial-angular relationships in light field images. When fusing information from multiple perspectives, they are prone to geometric distortion or loss of details.

[0004] Among deep learning-based methods, Convolutional Neural Networks (CNNs) are widely used in image denoising tasks due to their powerful local feature extraction capabilities. However, CNNs have limited receptive fields, making it difficult to capture long-distance dependencies between multiple viewpoints in light field images, and they are prone to insufficient viewpoint consistency when processing large-view light fields. Although the Transformer architecture can model global dependencies through self-attention mechanisms, its ability to capture local texture details is weak, and its computational complexity is high, making it difficult to directly apply to real-time processing of high-resolution light field images.

[0005] Therefore, how to construct a light field image denoising method that can effectively capture local texture details, model global space-angle dependencies, maintain the integrity of the light field geometry, and reduce computational complexity has become a pressing technical problem in this field. Summary of the Invention

[0006] To address the issues of information attenuation and high computational complexity in existing light field image denoising techniques for multi-view and multi-scale processing, this disclosure provides a light field image denoising method and apparatus based on a dual-branch hybrid architecture. Through an innovative design that integrates convolutional neural networks and Transformer architecture, efficient denoising is achieved while maintaining the integrity of the light field geometry.

[0007] According to a first aspect of this disclosure, a light field image denoising method based on a dual-branch hybrid architecture is provided, comprising: The original light field image is preprocessed to generate original-noise image pairs, and the original-noise image pairs are used as training datasets. A dual-branch hybrid architecture light field image denoising network model is constructed, comprising a local modeling branch, a global modeling branch, a feature fusion module, and a progressive multi-scale reconstruction module; wherein... The local modeling branch uses a multi-scale feature decoupling mechanism to extract spatial-angle local features, while the global modeling branch uses an angular distance-aware attention mechanism to model global spatial-angle dependencies. The feature fusion module is used to fuse the features output by the local and global modeling branches. The progressive multi-scale reconstruction module restores image details through residual learning and cross-scale aggregation. A progressive training strategy is adopted to train the light field image denoising network model based on the training dataset, thereby obtaining a trained light field image denoising network model. The noisy light field image to be processed is input into the trained light field image denoising network model, and the reconstructed denoised image is output.

[0008] In a preferred embodiment, the preprocessing of the original light field image to generate original-noise image pairs, and the use of the original-noise image pairs as a training dataset, includes: The original light field image is recombined into a spatial array containing multiple sub-viewpoints according to the sub-aperture image format; Each sub-view image in the spatial array is cropped to a predetermined pixel size to fit the network input, and training samples are generated using a preset pixel overlap strategy; Based on the training samples, noise within a preset intensity range is introduced during the training phase to construct original-noise image pairs.

[0009] In a preferred embodiment, the local modeling branch extracts spatial-angular local features through a multi-scale feature decoupling mechanism, including: After converting the light field sub-aperture image into a light field macro-pixel image, spatial features, angular features, and spatial-angular interaction features are extracted sequentially through a multi-scale feature decoupling mechanism. The spatial features, angular features, and spatial-angular interaction features are dynamically fused by generating weighting coefficients through global average pooling and the Softmax function to obtain CNN feature representations. After feature extraction at each layer, downsampling is performed to construct a multi-scale feature map, and the semantic expression capability is improved through a multi-scale context distillation module.

[0010] In a preferred embodiment, the spatial features are extracted through dilated convolution, wherein the kernel size of the dilated convolution is a preset size, and the dilation rate is related to the number of angles and viewpoints. The angular features are extracted through convolution of a preset size, and the perspective is perceived through non-overlapping sampling with a stride related to the number of angle viewpoints. The spatial-angular interaction features are extracted through asymmetric convolution.

[0011] In a preferred embodiment, the global modeling branch employs a pyramid structure for multi-level downsampling, specifically including EPI sub-branches and spatial sub-branches, wherein... The EPI sub-branch models the space-angle relationship based on an angular distance-aware attention mechanism; The spatial sub-branch, combined with a spatial attention mechanism, enhances global perception. The features output by the EPI sub-branch and the spatial sub-branch are automatically weighted through a gating fusion mechanism.

[0012] In a preferred embodiment, the feature fusion module is used to fuse the features output by the local modeling branch and the global modeling branch, including: Based on the EPI features in the global modeling branch as geometric priors, the texture features extracted from the local modeling branch are spatially aligned using a cross-attention mechanism; Convolution operations are used to predict sampling point offsets, and deformable convolution is introduced to compensate for cross-view geometric misalignment in order to improve pixel-level structural consistency. The image reconstruction quality is improved by enhancing salient regions of features through a dual-channel attention mechanism at both the spatial and channel levels.

[0013] In a preferred embodiment, the progressive multi-scale reconstruction module restores image details through the following steps: After upsampling at each level using sub-pixel level, the current level feature map is concatenated with the feature passed from the previous level to obtain the fused feature. A feature decoding block is introduced to decode the fused features to obtain deep semantic features, mid-level structural features and shallow texture features. The feature decoding block includes a convolutional layer, an activation function and a channel attention module, and the decoding process uses residual representation to preserve the original feature information. A cross-scale feature aggregation strategy is adopted to progressively fuse the deep semantic features, mid-level structural features, and shallow texture features; The number of channels in the progressively fused features is adjusted by convolution operations, and residual connections are performed with the input noisy image to output the final denoised image.

[0014] As a preferred implementation, a progressive training strategy is adopted. When training the light field image denoising network model based on the training dataset, supervision signals are gradually introduced in order from low resolution to high resolution, specifically including: In the early training stage, the low-resolution feature map of the original light field image is used as input, and the global structural features of the light field are learned based on the reconstruction loss. In the mid-training stage, a feature map of medium resolution in the original light field image is introduced. Based on the global structural feature learning, supervision of the mid-level structural features is added. The mid-level structural features include image edge contours, regional texture distribution, and local geometric correspondence between viewpoints. The structural differences between adjacent viewpoint sub-images are constrained by angle consistency loss. In the later stages of training, feature maps with the original resolution of the original light field image are used for training to learn detailed textures and high-frequency information. The gradient distribution difference between the reconstructed image and the original image in the EPI domain is constrained by the EPI gradient loss.

[0015] In a preferred embodiment, when training the light field image denoising network model using the training dataset with a progressive training strategy, the method further includes weighted joint optimization using multiple loss functions, wherein the expression for the multiple loss functions is: ; Among them, the Indicates the total loss. To rebuild the losses, For loss of angular consistency, For EPI gradient loss, In order to perceive loss, , , , These are the weighting factors for each loss term.

[0016] According to a second aspect of this disclosure, a light field image denoising apparatus based on a dual-branch hybrid architecture is provided, comprising: The data preprocessing module is used to preprocess the original light field images in the historical dataset to generate original-noise image pairs, and use the original-noise image pairs as the training dataset. A light field image denoising network model construction module is provided, which constructs a dual-branch hybrid architecture light field image denoising network model. The network model includes a local modeling branch, a global modeling branch, a feature fusion module, and a progressive multi-scale reconstruction module. The local modeling branch uses a multi-scale feature decoupling mechanism to extract spatial-angle local features, while the global modeling branch uses an angular distance-aware attention mechanism to model global spatial-angle dependencies. The feature fusion module is used to fuse the features output by the local and global modeling branches. The progressive multi-scale reconstruction module restores image details through residual learning and cross-scale aggregation. The light field image denoising network model training module is used to train the light field image denoising network model according to the training dataset using a progressive training strategy, so as to obtain a trained light field image denoising network model. The denoised image output module is used to input the noisy light field image to be processed into the trained light field image denoising network model and output the reconstructed denoised image.

[0017] According to a third aspect of this disclosure, an electronic device is provided, comprising at least one processor and a memory communicatively connected to said at least one processor; wherein the memory stores a computer program executable by said at least one processor, said computer program being executed by said at least one processor to enable said at least one processor to perform the method as described in any of the preceding claims.

[0018] Compared with the prior art, this disclosure achieves the following beneficial effects: (1) The light field image denoising method proposed in this invention is based on a dual-branch hybrid architecture of CNN and Transformer, which can effectively take into account both local texture information modeling and global structure modeling, and significantly improve the denoising quality while maintaining the geometric consistency of the light field. First, by uniformly cropping the sub-aperture and injecting noise into the light field image, a high-quality dataset of original noisy image pairs is constructed to ensure that the network has good generalization ability under different noise intensities.

[0019] (2) The network model proposed in this invention integrates the spatial angle detail extraction capability of the CNN branch with the global dependency modeling advantage of the Transformer branch, thus avoiding the problem of insufficient performance of traditional convolutional networks in large-view light fields. The multi-scale context distillation module in the CNN branch significantly improves the expressive power of structural features at different scales, while the Transformer branch enhances the modeling ability of multi-view geometric constraints by introducing an angular distance-aware attention mechanism and an image pyramid structure.

[0020] (3) In the feature fusion stage, this invention introduces deformable convolution and dual-channel spatial attention mechanisms, which significantly improves the cross-modal feature alignment effect and effectively alleviates the problems of viewpoint drift and structural misalignment. In the image reconstruction stage, a progressive multi-scale upsampling and residual decoding strategy is adopted to enable the model to have stronger detail restoration capabilities. (4) During the training process, this invention also introduces joint constraints of angle consistency loss and EPI gradient loss, which enhances the coherence of the internal structure of the light field and the consistency of the viewpoint, ensuring the dual advantages of the generated image in terms of texture fidelity and angle coordination. This invention is particularly suitable for high-noise light field image scenes, and its denoising effect is better than that of existing mainstream methods. It has good application prospects in actual sensor imaging and low-light scenes.

[0021] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0022] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A flowchart of a light field image denoising network method based on a dual-branch hybrid architecture according to an embodiment of the present disclosure is shown; Figure 2 A structural diagram of the denoising model according to an embodiment of this disclosure is shown; Figure 3 A structural diagram of the light field image feature decoupling module in a CNN branch according to an embodiment of this disclosure is shown; Figure 4 A structural diagram of spatially and angularly separable (SAS) Transformer Blocks in a Transformer branch according to an embodiment of this disclosure is shown. Figure 5 A structural diagram of the Feature Fusion Module (FFM) according to an embodiment of this disclosure is shown; Figure 6 A structural diagram of the progressive multiscale reconstruction module according to an embodiment of this disclosure is shown; Figure 7 A basic comparison diagram of the denoising effect of a test set of light field images with Gaussian white noise according to embodiments of the present disclosure is shown; Figure 8 An extended diagram of the multi-scene denoising effect of a light field image test set with Gaussian white noise according to an embodiment of this disclosure is shown. Figure 9 A detailed method comparison diagram of a light field image test set with Gaussian white noise, according to embodiments of this disclosure, is shown. Figure 10 A block diagram of a light field image denoising network device based on a dual-branch hybrid architecture according to an embodiment of the present disclosure is shown; Figure 11A schematic diagram of an exemplary electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0024] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0025] This embodiment provides a light field image denoising method based on a dual-branch hybrid architecture. By combining the advantages of CNN and Transformer, it solves the problems of local detail loss, global geometric distortion and low computational efficiency in multi-view modeling of traditional methods.

[0026] like Figure 1 The diagram shows a flowchart of a light field image denoising network method based on a dual-branch hybrid architecture according to an embodiment of this disclosure. The method 100 includes: S110: Preprocess the original light field image to generate original-noise image pairs, and use the original-noise image pairs as training datasets.

[0027] In some embodiments, preprocessing the original light field image includes the following steps: First, the original light field image is reorganized into a spatial array containing multiple sub-viewpoints according to the sub-aperture image format; Secondly, in order to enhance the spatial continuity and structural coherence between adjacent sub-view images, each sub-view image in the spatial array is cropped to a predetermined pixel size to adapt to the network input, and a preset pixel overlap strategy is used to generate training samples. Finally, to ensure that the network can exhibit good robustness under different noise intensities, noise with a preset intensity range is introduced into the training phase based on the training samples formed above to construct original-noise image pairs.

[0028] For example, firstly, the original light field images in the historical dataset are organized in the Sub-Aperture Image (SAI) format, dividing each frame of the light field image into a 5×5 sub-view array, generating a total of 25 sub-view images. Secondly, to enhance the spatial continuity and structural coherence between adjacent sub-view images, each sub-view image is cropped to a 64×64 pixel image, while a 32-pixel overlap cropping strategy is adopted to ensure that there is a certain degree of regional overlap between sub-images, thereby better preserving the contextual information of adjacent view images during network training. Finally, to ensure that the network can exhibit good robustness under different noise intensities, a standard deviation is added to each training image. Gaussian noise is used to generate noisy image pairs, which serve as input for network training. It's worth noting that the addition of Gaussian noise simulates common noise interference encountered during sensor acquisition, image transmission, and processing. This step allows the model to adapt to different noise environments, improving its generalization ability in practical applications.

[0029] S120: Construct a dual-branch hybrid architecture light field image denoising network model, which includes a local modeling branch, a global modeling branch, a feature fusion module, and a progressive multi-scale reconstruction module. The local modeling branch uses a multi-scale feature decoupling mechanism to extract spatial-angular local features, the global modeling branch uses an angular distance-aware attention mechanism to model global spatial-angular dependencies, the feature fusion module is used to fuse the features output by the local modeling branch and the global modeling branch, and the progressive multi-scale reconstruction module restores image details through residual learning and cross-scale aggregation.

[0030] In some embodiments, the dual-branch hybrid architecture light field image denoising network model constructed in this disclosure is as follows: Figure 2 As shown, a parallel structure is formed by a local modeling branch (CNN branch) and a global modeling branch (Transformer branch), which efficiently model the local texture and global spatial dependency features of the image, respectively. Then, through the joint action of subsequent fusion and reconstruction modules, effective denoising and detail restoration of the light field image are achieved.

[0031] like Figure 2 As shown, the CNN branch sequentially extracts spatial features, angular features, and spatial-angular local features (EPI features), and generates weighted coefficients through global average pooling and the Softmax function to dynamically fuse the extracted spatial features, angular features, and spatial-angular interaction features. Furthermore, after feature extraction at each layer, a downsampling operation is applied to construct a multi-scale feature map, while a multi-scale context distillation module enhances semantic expressive power.

[0032] Spatial features are extracted through dilated convolution, with the kernel size of the dilated convolution being a preset size, and the dilation rate being related to the number of angles and viewpoints. Angle features are extracted through convolution of a preset size, and viewpoint perception is achieved through non-overlapping sampling with a stride related to the number of angles and viewpoints. Spatial-angle interaction features are extracted through asymmetric convolution.

[0033] The Transformer branch employs a pyramid structure for K-level downsampling, consisting of an EPI branch and a spatial branch. The EPI branch models the spatial-angular relationship based on an angular distance-aware attention mechanism, while the spatial branch enhances global perception by incorporating a spatial attention mechanism. The features output from the two branches are fused through gating and then aligned and recalibrated with the features output from the CNN branch in the feature fusion module. Finally, the features are input into the progressive multi-scale reconstruction module, where high-quality image details are recovered through progressive upsampling and convolutional decoding.

[0034] For example, such as Figure 3 As shown, the CNN branch takes the rearranged macropixel image (MacPI) as input. The core of this branch is the "light field feature decoupling module (Distg_Blocks)", which is used to structurally decompose the light field image in space, angle and space-angle mixed dimensions, thereby realizing the full modeling and reconstruction of multi-dimensional features.

[0035] First, the input light field image is organized into a sub-aperture image (SAI) array, and then transformed into a macropixel image (MacPI) through a rearrangement operation, which maps A×A viewpoint images to different regions of a single planar image. This operation facilitates parallel processing of multiple sub-viewpoint information in a convolutional manner.

[0036] Three types of decoupling features are extracted sequentially from the MacPI image: Spatial features Dilated convolution (DConv) is used with a 3×3 kernel and a dilation rate set to the number of viewpoints A to expand the receptive field and capture fine-grained textures. The spatial feature extraction formula is as follows: ; Angular features By using an A×A convolution kernel and setting a stride of A, non-overlapping sampling is achieved, enhancing the perceptual ability across viewpoint dimensions and modeling the structural correspondence between sub-viewpoints. The angle feature extraction formula is: ; Spatial-angle interaction features : Using 1× The asymmetric convolution with a stride of (1, A) is used to jointly model the coupling relationship between the image space and the view dimension. The formula for space-view interaction feature extraction is: ; To achieve the above spatial features Angular features and spatial-angle interaction features To address the fusion and weighting of input features, this disclosure proposes an adaptive weighted fusion mechanism. Firstly, it performs fusion and weighting on the input features... Perform global average pooling (GAP) and then input it into fully connected layers (FC) to generate three weight coefficients. The final weight combination is obtained through Softmax normalization: ; The final weighted summation yields the fused feature representation. : ; It is worth mentioning that, to enhance the network's ability to express complex scenes, a multi-scale context distillation (MCD) module is embedded in each feature extraction stage of the CNN branch. This module uses depthwise separable convolutions (DSCs) with different dilation rates ([1, 6, 12]) to achieve the aggregation of semantic information across scales. This mechanism introduces different receptive fields through parallel convolutional paths, thereby improving the network's ability to model semantics at different scales. After each feature fusion and distillation process, the network performs a downsampling operation, typically using a 3×3 convolution with a stride of 2 to halve the feature map size and increase the number of channels. This helps to gradually build deep semantic feature maps and form a K-level pyramid structure for subsequent alignment and fusion with the Transformer branch at different scales.

[0037] For example, such as Figure 4 As shown, the Transformer branch, which runs parallel to the CNN branch, aims to model global dependencies and geometric consistency in light field images. It achieves joint modeling of global structural information and angular consistency by introducing a spatial-angle separation structure (SAS Transformer).

[0038] The Transformer branch employs a pyramid structure for feature construction, coupled with a multi-scale attention module to achieve refined cross-view modeling capabilities. First, the input light field image features are downsampled sequentially through three consecutive convolution operations with a stride of 2, gradually constructing a K-layer image pyramid representation. Each pyramid feature map corresponds to image information at a specific scale, with resolutions of original image, 1 / 2, 1 / 4, etc. This structure effectively captures image pattern features at different resolutions while improving adaptability to low-frequency and high-frequency noise. Each feature map is fed into a spatially and angularly separable Transformer module (SAS Transformer Block), which uses axial decomposition to reconstruct the macropixel image (MacPI) into three geometric representations: Horizontal EPI—slicing along the UW plane, maintaining the association between spatial dimension W and angular dimension V; Vertical EPI—slicing along the VH plane, maintaining the association between spatial dimension H and angular dimension U; and Spatial View—maintaining the original arrangement to preserve global context. Each module consists of three parts: a horizontal EPI sub-branch, a vertical EPI sub-branch, and a spatial sub-branch, which focus on geometric angle consistency modeling and global context modeling, respectively.

[0039] In the EPI sub-branch, the input feature map is rearranged into Epipolar PlaneImage (EPI) slices along the horizontal or vertical direction, representing a set of quasi-1D sub-view sequences. A linear transformation is performed on each slice to obtain the three tensors required for the self-attention mechanism: the query matrix Q, the key matrix K, and the value matrix V. The formula is as follows: ; Then, the EPI sub-branch uses an Angular Distance Aware Multi-Head Attention (ADA-MHA) module to perform weighted attention modeling on the above Q / K / V tensors. This mechanism introduces a prior angular distance matrix on the basis of traditional multi-head attention. ∈RA×A, explicitly modeling the geometric dependencies between sub-views. Its attention formula is as follows: ; Where Softmax represents the normalization function for a finite number of discrete probability distributions. Indicate the dimensions of the Q matrix and the K matrix. This is the angular distance matrix, with values ​​predefined based on the positional relationships between sub-viewpoints. This attention mechanism effectively enhances the model's ability to perceive angular structural misalignments, improving the modeling effect of angular consistency.

[0040] The spatial sub-branch takes the downsampled image feature map as input and processes it using a standard Multi-Head Self Attention (MHSA) module to extract long-range contextual information. Compared to the CNN branch, the spatial sub-branch is better at global perception and texture structure restoration, helping to compensate for the inadequacy of local features in capturing global semantic information. In each SAS Transformer Block, the feature outputs from the EPI sub-branch and the spatial sub-branch are integrated through a gated fusion mechanism. This mechanism introduces learnable gate weights to automatically adjust the feature importance of the two sub-branches, thereby dynamically balancing global structural understanding and angular geometry modeling. The feature maps output from all K-level SAS modules are finally aggregated layer by layer and fed into a unified feature fusion module along with the feature maps extracted by the CNN branch, achieving effective integration across multiple scales, sources, and semantic dimensions, providing full information support for subsequent image restoration and reconstruction.

[0041] In some embodiments, such as Figure 5 As shown, the feature fusion module is used to fuse the features output by the local modeling branch and the global modeling branch, including the following steps: First, the Feature Fusion Module (FFM) uses the EPI features in the Transformer branch as a geometric prior and performs spatial alignment of the texture features extracted by the CNN branch through a cross-attention mechanism. Then, it uses 3×3 convolution to predict the sampling point offset and introduces deformable convolution to compensate for cross-view misalignment, thereby improving pixel-level structural consistency. Finally, it completes the enhancement of salient regions at both the spatial and channel levels through a dual-channel attention mechanism, thereby improving the reconstruction quality.

[0042] For example, firstly, features extracted from the EPI sub-branch in the Transformer branch. As a geometric prior, it guides the extraction of spatial texture features from CNN branches. Perform spatial alignment. The specific formula is as follows: ; in, This indicates a splicing operation along the channel dimension; The output features of the main branch of the Transformer. It is a depthwise separable convolution used to reduce computational complexity and extract fine texture structures.

[0043] After attention alignment, to address the geometric misalignment caused by baseline distance between different viewpoints in the light field, a deformable convolution mechanism is further introduced for pixel-level structural compensation. The calculation formula is as follows:

[0044] in, This represents the aligned feature map output after a deformable convolution operation. It geometrically offsets the input features in space to better align with the spatial structure of the target features. DeformConv dynamically predicts the offset at each convolution kernel position. Represents the input feature map, It represents the spatial offset of each convolution position in deformable convolution, reflecting the pixel-level misalignment between the target feature and the input feature.

[0045] After completing spatial alignment and geometric correction, a dual-channel attention mechanism combining spatial and channel levels is introduced to significantly enhance key region features and further improve the response of key regions.

[0046] Among them, Channel Attention (CA) is used to enhance input features. First, channel context information is extracted using a 3×3 convolution, and then channel importance weights are obtained using a Sigmoid activation function. The formula is as follows: ; Spatial attention enhancement (SA): incorporating input features A spatial attention map is formed by capturing salient regions at the pixel level through 1×1 convolution and then normalizing the data using a sigmoid function. The formula is as follows: ; The final output fused feature is given by the following formula: .

[0047] In some embodiments, such as Figure 6 As shown, in the final stage of the light field image denoising network, a progressive multi-scale reconstruction module is introduced to restore high-frequency details and spatial consistency features of the image. The design goal of this module is to achieve a coarse-to-fine image reconstruction process, balancing resolution improvement and texture preservation.

[0048] First, the features fused from the CNN and Transformer branches are input to the progressive multi-scale reconstruction module. To avoid the checkerboard artifacts introduced by traditional transposed convolution, the module employs a PixelShuffle operation to achieve sub-pixel-level upsampling, progressively increasing the image resolution. After each upsampling stage, the feature map at the current resolution is concatenated with the upsampled features from the previous layer. Subsequently, a feature decoding block is introduced to restore details of the concatenated result, yielding deep semantic features, mid-level structural features, and shallow texture features.

[0049] Each feature decoding block consists of the following structure: a 3×3 standard convolutional layer, a LeakyReLU activation function, and a coordinate attention (CA) mechanism module to enhance spatial location sensitivity and channel saliency.

[0050] The feature decoding process is expressed in the following residual form: ; in, Indicates the current input features. This represents the output features after decoding. The introduction of residual structures helps to preserve key information from the original features, improving reconstruction stability and convergence speed.

[0051] After multi-level features are decoded step-by-step through the above process, a cross-scale feature aggregation strategy is employed to fuse semantics at different levels. The fusion strategy follows the order of "deep semantic features - mid-level structural features - shallow texture features," integrating multi-scale semantic information from bottom to top to improve image texture expressiveness and structural clarity. Finally, a 1×1 convolution operation is used to adjust the number of channels to match the input image, and a residual connection is performed with the original noisy image to output the final denoising result. The formula is as follows: ; in, This represents the image after denoising. The original input image contains noise.

[0052] S130: Using a progressive training strategy, the light field image denoising network model is trained based on the training dataset to obtain a trained light field image denoising network model. In some embodiments, to progressively optimize model performance from coarse to fine, the light field image denoising network employs a progressive training strategy. In the initial training stage, low-resolution feature maps of the original light field image are used as input, and global structural features of the light field are learned based on reconstruction loss. In the middle training stage, medium-resolution feature maps of the original light field image are introduced, and supervision of the mid-level structural features is added based on the global structural feature learning. The mid-level structural features include image edge contours, regional texture distribution, and local geometric correspondences between viewpoints. The structural differences between adjacent viewpoint sub-images are constrained by angle consistency loss. In the later training stage, the original resolution feature maps of the original light field image are used for training to learn detailed textures and high-frequency information. The gradient distribution difference between the reconstructed image and the original image in the EPI domain is constrained by EPI gradient loss.

[0053] For example, in the early stages of training, the model focuses on learning low-resolution and global structural information to quickly establish a perception of the basic content of the image. Subsequently, high-resolution and detail-level supervision signals are gradually introduced to improve the model's ability to recover textures and edges, ensuring the collaborative learning of multi-scale features. To enhance the consistency of the light field viewpoint, an Angular Consistency Loss is introduced during training. This loss constrains the structural coherence between adjacent sub-view images, maintaining geometric consistency between viewpoints. Its mathematical formula is as follows: ; in, and These represent the i-th and j-th sub-view images output by the network, respectively. and For the corresponding noise-free image, This represents the set of adjacent viewpoints. This loss encourages the network to preserve the relative structural relationships between viewpoints while reducing noise.

[0054] To preserve the linear structure features in the light field image, EPI gradient loss is introduced to protect the detail information along the viewpoint axis from excessive smoothing. Its definition is... as follows: ; in, and These are EPI slices of the reconstructed image and the reference image in view coordinates (u,v), respectively. This represents the gradient operator in the horizontal direction. This loss function preserves the sharpness of the edges of the light field structure in the image by contrasting the gradient, and it is particularly effective in protecting geometric boundaries and textures.

[0055] In summary, the final multi-loss function expression for the training objective function in this embodiment is: ; Among them, the Indicates the total loss. To rebuild the losses, For loss of angular consistency, For EPI gradient loss, In order to perceive loss, , , , The weighting factors for each loss term can be optimized through experiments.

[0056] S140: Input the noisy light field image to be processed into the trained light field image denoising network model, and output the reconstructed denoised image.

[0057] In some embodiments, the experimental parameters and training sets used in the experiments of this disclosure are as follows: EPFL, HCI_new, HCI_old, INRIA_Lytro, and Stanford_Gantry datasets are used as training and testing sets, respectively. The performance of the present invention is evaluated using objective evaluation metrics PSNR and SSIM. PSNR measures the denoising effect of the image denoising model; a higher PSNR value indicates a better denoising effect. SSIM measures the similarity between two images; the maximum value of SSIM is 1, and a higher value indicates a higher similarity between the two images. The formulas for PSNR and SSIM are: ; Where MSE represents the mean squared error between the original image and the reconstructed image, and MAXI represents the maximum possible pixel value of the original image. It is the mean of the image x. It is the mean of the image y. It is the variance of the image x. It is the variance of the image y. It is the covariance of image x and image y. and It is a constant that maintains stability.

[0058] Experimental Results Analysis: Noise standard deviations were added to the five test sets: EPFL, HCI_new, HCI_old, INRIA_Lytro, and Stanford_Gantry. Gaussian noise was eliminated by using a pre-trained light field image denoising network model to predict noise-free images, and the PSNR and SSIM values ​​were calculated. Table 1 shows the denoising results for some images in the five test sets. Figure 7 As shown.

[0059] Table 1

[0060] Among them, Hyperfan4D, LFBM5D, APA, DRLF, MSPnet, and LF-SRS are all light field image denoising algorithms.

[0061] Experiments show that the dual-branch hybrid architecture of CNN and Transformer adopted in this disclosure outperforms other existing mainstream methods in both PSNR and SSIM, especially in high-noise (σ>30) scenarios.

[0062] According to the above embodiments of this disclosure, the following technical effects are achieved: (1) The light field image denoising method proposed in this invention is based on a dual-branch hybrid architecture of CNN and Transformer, which can effectively take into account both local texture information modeling and global structure modeling, and significantly improve the denoising quality while maintaining the geometric consistency of the light field. First, by uniformly cropping the sub-aperture and injecting noise into the light field image, a high-quality dataset of original noisy image pairs is constructed to ensure that the network has good generalization ability under different noise intensities.

[0063] (2) The network model proposed in this invention integrates the spatial angle detail extraction capability of the CNN branch with the global dependency modeling advantage of the Transformer branch, thus avoiding the problem of insufficient performance of traditional convolutional networks in large-view light fields. The multi-scale context distillation module in the CNN branch significantly improves the expressive power of structural features at different scales, while the Transformer branch enhances the modeling ability of multi-view geometric constraints by introducing an angular distance-aware attention mechanism and an image pyramid structure.

[0064] (3) In the feature fusion stage, this invention introduces deformable convolution and dual-channel spatial attention mechanisms, which significantly improves the cross-modal feature alignment effect and effectively alleviates the problems of viewpoint drift and structural misalignment. In the image reconstruction stage, a progressive multi-scale upsampling and residual decoding strategy is adopted to enable the model to have stronger detail restoration capabilities. (4) During the training process, this invention also introduces joint constraints of angle consistency loss and EPI gradient loss, which enhances the coherence of the internal structure of the light field and the consistency of the viewpoint, ensuring the dual advantages of the generated image in terms of texture fidelity and angle coordination. This invention is particularly suitable for high-noise light field image scenes, and its denoising effect is better than that of existing mainstream methods. It has good application prospects in actual sensor imaging and low-light scenes.

[0065] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0066] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0067] Figure 10 A block diagram of a light field image denoising network device based on a dual-branch hybrid architecture according to an embodiment of the present disclosure is shown. Figure 10 As shown, the device 800 includes: The data preprocessing module 810 is used to preprocess the original light field images in the historical dataset to generate original-noise image pairs, and use the original-noise image pairs as training datasets. The light field image denoising network model construction module 820 constructs a dual-branch hybrid architecture light field image denoising network model, which includes a local modeling branch, a global modeling branch, a feature fusion module, and a progressive multi-scale reconstruction module; wherein... The local modeling branch uses a multi-scale feature decoupling mechanism to extract spatial-angle local features, while the global modeling branch uses an angular distance-aware attention mechanism to model global spatial-angle dependencies. The feature fusion module is used to fuse the features output by the local and global modeling branches. The progressive multi-scale reconstruction module restores image details through residual learning and cross-scale aggregation. The light field image denoising network model training module 830 is used to train the light field image denoising network model according to the training dataset using a progressive training strategy, so as to obtain a trained light field image denoising network model. The denoised image output module 840 is used to input the noisy light field image to be processed into the trained light field image denoising network model and output the reconstructed denoised image.

[0068] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0069] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0070] Figure 11 A schematic block diagram of an electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0071] Electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in ROM 902 or a computer program loaded into RAM 903 from storage unit 908. RAM 903 can also store various programs and data required for the operation of electronic device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. I / O interface 905 is also connected to bus 904.

[0072] Multiple components in electronic device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of displays, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows electronic device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0073] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as method 100. For example, in some embodiments, method 100 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform method 100 by any other suitable means (e.g., by means of firmware).

[0074] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0075] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0076] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0077] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including voice input, speech input, or tactile input).

[0078] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0079] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0080] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0081] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A light field image denoising network method based on a dual-branch hybrid architecture, characterized in that, include: The original light field image is preprocessed to generate original-noise image pairs, and the original-noise image pairs are used as training datasets. A dual-branch hybrid architecture light field image denoising network model is constructed, comprising a local modeling branch, a global modeling branch, a feature fusion module, and a progressive multi-scale reconstruction module; wherein... The local modeling branch uses a multi-scale feature decoupling mechanism to extract spatial-angle local features. The global modeling branch models global spatial-angle dependencies using an angular distance-aware attention mechanism. The global modeling branch employs a pyramid structure for multi-level downsampling, specifically including an EPI sub-branch and a spatial sub-branch. The EPI sub-branch models spatial-angle relationships based on the angular distance-aware attention mechanism; the spatial sub-branch enhances global awareness by incorporating a spatial attention mechanism. The features output by the EPI and spatial sub-branch are automatically weighted through a gating fusion mechanism. The feature fusion module is used to fuse the features output by the local and global modeling branches, including: Based on the EPI features in the global modeling branch as geometric priors, the texture features extracted from the local modeling branch are spatially aligned using a cross-attention mechanism; the sampling point offset is predicted using convolution operations, and deformable convolution is introduced to compensate for cross-view geometric misalignment to improve pixel-level structural consistency; the features are enhanced in salient regions through a dual-channel attention mechanism at both spatial and channel levels to improve image reconstruction quality; the progressive multi-scale reconstruction module restores image details through residual learning and cross-scale aggregation. A progressive training strategy is adopted, and the light field image denoising network model is trained on the training dataset to obtain a trained light field image denoising network model. During the training of the light field image denoising network model on the training dataset, supervision signals are gradually introduced in order from low resolution to high resolution, specifically including: In the early training stage, the low-resolution feature map of the original light field image is used as input, and the global structural features of the light field are learned based on the reconstruction loss. In the mid-training stage, a feature map of medium resolution in the original light field image is introduced. Based on the global structural feature learning, supervision of the mid-level structural features is added. The mid-level structural features include image edge contours, regional texture distribution, and local geometric correspondence between viewpoints. The structural differences between adjacent viewpoint sub-images are constrained by angle consistency loss. In the later stages of training, feature maps with the original resolution of the original light field image are used for training to learn detailed textures and high-frequency information. The gradient distribution difference between the reconstructed image and the original image in the EPI domain is constrained by the EPI gradient loss. The noisy light field image to be processed is input into the trained light field image denoising network model, and the reconstructed denoised image is output.

2. The method according to claim 1, characterized in that, The preprocessing of the original light field image to generate original-noise image pairs, and the use of these original-noise image pairs as training datasets, includes: The original light field image is recombined into a spatial array containing multiple sub-viewpoints according to the sub-aperture image format; Each sub-view image in the spatial array is cropped to a predetermined pixel size to fit the network input, and training samples are generated using a preset pixel overlap strategy; Based on the training samples, noise within a preset intensity range is introduced during the training phase to construct original-noise image pairs.

3. The method according to claim 2, characterized in that, The local modeling branch extracts spatial-angular local features through a multi-scale feature decoupling mechanism, including: After converting the light field sub-aperture image into a light field macro-pixel image, spatial features, angular features, and spatial-angular interaction features are extracted sequentially through a multi-scale feature decoupling mechanism. The spatial features, angular features, and spatial-angular interaction features are dynamically fused by generating weighting coefficients through global average pooling and the Softmax function to obtain CNN feature representations. After feature extraction at each layer, downsampling is performed to construct a multi-scale feature map, and the semantic expression capability is improved through a multi-scale context distillation module.

4. The method according to claim 3, characterized in that, The spatial features are extracted through dilated convolution, where the kernel size of the dilated convolution is a preset size, and the dilation rate is related to the number of angles and viewpoints. The angular features are extracted through convolution of a preset size, and the perspective is perceived through non-overlapping sampling with a stride related to the number of angle viewpoints. The spatial-angular interaction features are extracted through asymmetric convolution.

5. The method according to claim 1, characterized in that, The progressive multi-scale reconstruction module restores image details through the following steps: After upsampling at each level using sub-pixel level, the current level feature map is concatenated with the feature passed from the previous level to obtain the fused feature. A feature decoding block is introduced to decode the fused features to obtain deep semantic features, mid-level structural features and shallow texture features. The feature decoding block includes a convolutional layer, an activation function and a channel attention module, and the decoding process uses residual representation to preserve the original feature information. A cross-scale feature aggregation strategy is adopted to progressively fuse the deep semantic features, mid-level structural features, and shallow texture features; The number of channels in the progressively fused features is adjusted by convolution operations, and residual connections are performed with the input noisy image to output the final denoised image.

6. The method according to claim 1, characterized in that, When training the light field image denoising network model using the progressive training strategy based on the training dataset, the method further includes weighted joint optimization using multiple loss functions, wherein the expression for the multiple loss functions is: ； Among them, the Indicates the total loss. To rebuild the losses, For loss of angular consistency, For EPI gradient loss, In order to perceive loss, , , , These are the weighting factors for each loss term.

7. A light field image denoising network device based on a dual-branch hybrid architecture, characterized in that, include: The data preprocessing module is used to preprocess the original light field images in the historical dataset to generate original-noise image pairs, and use the original-noise image pairs as the training dataset. A light field image denoising network model construction module is provided, which constructs a dual-branch hybrid architecture light field image denoising network model. The network model includes a local modeling branch, a global modeling branch, a feature fusion module, and a progressive multi-scale reconstruction module. The local modeling branch uses a multi-scale feature decoupling mechanism to extract spatial-angle local features. The global modeling branch models global spatial-angle dependencies using an angular distance-aware attention mechanism. The global modeling branch employs a pyramid structure for multi-level downsampling, specifically including an EPI sub-branch and a spatial sub-branch. The EPI sub-branch models spatial-angle relationships based on the angular distance-aware attention mechanism; the spatial sub-branch enhances global awareness by incorporating a spatial attention mechanism. The features output by the EPI and spatial sub-branch are automatically weighted through a gating fusion mechanism. The feature fusion module is used to fuse the features output by the local and global modeling branches, including: Based on the EPI features in the global modeling branch as geometric priors, the texture features extracted from the local modeling branch are spatially aligned using a cross-attention mechanism; the sampling point offset is predicted using convolution operations, and deformable convolution is introduced to compensate for cross-view geometric misalignment to improve pixel-level structural consistency; the features are enhanced in salient regions through a dual-channel attention mechanism at both spatial and channel levels to improve image reconstruction quality; the progressive multi-scale reconstruction module restores image details through residual learning and cross-scale aggregation. The light field image denoising network model training module is used to train the light field image denoising network model using a progressive training strategy based on the training dataset, resulting in a trained light field image denoising network model. During training, the supervision signal is introduced progressively from low resolution to high resolution, specifically including: In the early training stage, the low-resolution feature map of the original light field image is used as input, and the global structural features of the light field are learned based on the reconstruction loss. In the mid-training stage, a feature map of medium resolution in the original light field image is introduced. Based on the global structural feature learning, supervision of the mid-level structural features is added. The mid-level structural features include image edge contours, regional texture distribution, and local geometric correspondence between viewpoints. The structural differences between adjacent viewpoint sub-images are constrained by angle consistency loss. In the later stages of training, feature maps with the original resolution of the original light field image are used for training to learn detailed textures and high-frequency information. The gradient distribution difference between the reconstructed image and the original image in the EPI domain is constrained by the EPI gradient loss. The denoised image output module is used to input the noisy light field image to be processed into the trained light field image denoising network model and output the reconstructed denoised image.

Images