Light field image denoising method
By combining local semantic scene classification and global scene type decision network, and dynamically allocating computing resources, the problems of regional difference processing and cross-platform adaptation in light field image denoising methods are solved, thereby improving denoising quality and efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA UNIV OF PETROLEUM (EAST CHINA)
- Filing Date
- 2026-04-16
- Publication Date
- 2026-07-21
AI Technical Summary
Existing light field image denoising methods fail to effectively distinguish the differences in denoising quality between different semantic regions, resulting in the loss of details in sensitive areas or over-processing of flat areas. At the same time, it is difficult to achieve cross-platform adaptation and real-time operation on devices with limited computing power.
By establishing a local semantic scene classification model to generate local scene heatmaps, constructing a global scene type local decision network and integrating a cross-attention mechanism, dynamically allocating computing resources to prioritize the denoising quality of high-priority areas, and using a Transformer architecture light field image denoising encoder for differentiated processing.
It achieves dynamic adjustment of denoising strategy according to scene type, improves denoising quality and achieves efficient adaptation on different devices, while maintaining robustness and interpretability.
Smart Images
Figure CN122434767A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, specifically to a method for denoising light field images. Background Technology
[0002] Existing light field image denoising methods (such as Gaussian filtering and BM3D) employ a uniform processing strategy for all regions, neglecting the differentiated sensitivity of different semantic regions (such as portraits and skies) to denoising quality. This leads to the loss of details in sensitive areas such as portraits or overprocessing of flat areas such as skies. While deep learning-based denoising methods can improve overall quality, they do not consider global scene context information (such as the difference in attention given to the same semantic category between family photos and landscape photos) and cannot dynamically adjust denoising strategies according to scene type. Existing methods lack effective computing resource allocation mechanisms on embedded devices or mobile terminals with limited computing power. Either the model is too large to run in real time, or uniform degradation leads to a severe decline in the quality of high-priority regions. Existing heterogeneous computing solutions usually require retraining models for different devices, making cross-platform adaptation difficult. Summary of the Invention
[0003] To address the aforementioned technical problems, this technical solution provides a method for denoising light field images and resolves the problems described above.
[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows: Light field image denoising methods include: S1. Based on several pre-set image databases, obtain several original image light field data, establish a local semantic scene classification model, analyze the local semantics of each image light field, output the scene type to which the local semantics of each image light field belongs, and obtain the local scene heat map of each image light field. S2. Establish a prior knowledge database of known scene types, construct a local decision network for global scene types, divide the local scene heatmaps of each image light field, and assign attention priority to the local semantic scene category of each image light field under the global scene type according to the local semantic-global scene attention mechanism. S3. Based on the transformer architecture, establish light field image denoising encoders for several scene types, build a priority resource heterogeneous encoder configuration model, and dynamically divide the local semantic scene category priority of each image light field under the global scene type according to the real-time available computing power resources of the device hardware to generate heterogeneous denoising tasks for the local semantic scene category of each image light field under the global scene type.
[0005] Preferably, step S1 specifically includes: Based on several pre-set image databases, several original image light field data are obtained. According to the sub-aperture image extraction, the central view and the neighboring view of the image light field are divided for early fusion and stacking. The pixel values of the divided images are normalized to obtain several image light field tensors. Based on several pre-set image databases, the LabelMe pixel annotation tool is used to create a central viewpoint label map according to the scene type of the central viewpoint pixels marked by manual annotation. The disparity map between the domain viewpoint and the central viewpoint label map is calculated to perform light field angle consistency mapping to obtain the domain viewpoint label map. Several image light field tensors are used as feature data, and the center viewpoint label map and the domain viewpoint label map are used as label data to construct the original training dataset for scene types. Based on DeepLabV3 semantic image segmentation encoder-decoder, a local semantic scene classification model is established. The original training dataset for scene types is used as input. The encoder convolutional block progressively downsamples several image light field tensors to extract local low-order object textures and local high-order object semantics of the image light field, and constructs a local multi-scale feature map of the image light field. By utilizing ASPP (Space Pyramid Pooling with Dilated Space) and employing multiple dilated convolutions with different dilation rates, the context information of local high-order semantic feature maps in the local multi-scale feature maps of the image light field is processed in parallel and then concatenated. The input two-dimensional convolution is used to progressively transpose the current local high-order semantic feature map, perform convolution upsampling, and concatenate the channel dimension of the output skip connection features of the convolution block to obtain the context-enhanced feature map of the image light field. Based on the decoder convolutional block, several output channels are established according to the total number of scene types. The probability vector distribution of each pixel position belonging to each scene type in the context enhancement feature map of the image light field is verified by the Softmax function. The scene type corresponding to the maximum probability of each pixel position belonging to each scene type is used as the hard label of the pixel, and the probability vector of each pixel position is used as the soft label to obtain the local semantic classification mask of each image light field. Create a blank matrix of the same size as the original image, traverse the semantic scene type label of each pixel position in the local semantic classification mask of the light field of each image, and assign it to the blank matrix of the same size as the original image with the same coordinates to obtain the local scene heatmap of the light field of each image.
[0006] Preferably, step S2 specifically includes: Based on several pre-defined image databases, using pre-labeled image data of known scene types, and according to maximum likelihood estimation, the local semantic scene category conditional probability, semantic co-occurrence conditional probability, and spatial layout conditional probability in the image data of known scene types are verified to obtain the local conditional probability distribution in the image data of known scene types and to establish a prior knowledge database of known scene types. Based on the local scene heatmaps of each image light field, we statistically analyze the histogram of the percentage of pixels of semantic scene categories in the local scene heatmap, the number of semantic scene category pairs of adjacent pixels, and the percentage of each type within the local space, and construct a global feature vector set for each image light field. Based on RandomForest, a decision tree for each scene type is constructed. The global feature vector set of the light field of each image is used as the root node, the prior knowledge of the known scene type is used as the branch node, and the probability distribution of each scene type is used as the leaf node. The maximum decrease of Gini impurity is used as the splitting target to build a local decision network for the global scene type. Based on the global scene type local decision network, according to the Bayesian posterior, read the scene type probability distribution stored in all leaf nodes, and observe the posterior probability of each scene type given the global feature vector of the light field of each image. Based on the local decision network of global scene type, the probability distribution of scene type stored in each leaf node is read as a conditional probability estimate of observing each scene type under a given global feature vector. The output probabilities of all decision trees are arithmetically averaged to obtain the posterior probability estimate of the global scene type pointing to the global feature vector of each image light field. The global scene type of each image light field is generated according to the maximum value of the posterior probability of the global scene type.
[0007] Preferably, step S2 further includes: Based on the cross-attention mechanism, a priority assignment function for the local semantic-global scene attention mechanism is constructed. The global scene type and local semantic scene category of each image light field are mapped to high-dimensional embedding vectors. According to the attention mechanism, the global scene type embedding vector is used as the query and the local semantic scene category embedding vector is used as the key to calculate the attention score between each local semantic scene category and the global scene type. Normalization is performed on the conditional probability estimates of each scene type observed under the given global feature vector. The prior importance weights of each semantic scene category of the global scene type are weighted and fused with the attention scores between each local semantic scene category and the global scene type to obtain the attention priority scores of the local semantic scene categories of each image light field under the global scene type.
[0008] Preferably, step S3 specifically includes: Based on the transformer architecture, according to the local semantic scene categories of each image light field under the global scene type, a multi-layer stacked Block structure light field image denoising encoder of several scene types is established. Several configuration processing levels are preset. The priority of the local semantic scene category of each image light field under the global scene type is used as the configuration processing level selection condition. The real-time callable computing power resources of the device hardware are used as the configuration processing level selection constraint boundary to build a priority resource heterogeneous encoder configuration model. Compared with the prior art, the beneficial effects of the present invention are as follows: This invention proposes a flowchart for a light field image denoising method. S1 establishes a local semantic scene classification model to generate a local scene heatmap, achieving a leap from "blind denoising" to "semantic-aware denoising." S2 constructs a global scene type local decision network and integrates prior knowledge and a cross-attention mechanism to achieve scene-adaptive differentiated priority assignment, making computational power allocation more consistent with human visual perception characteristics. S3 establishes a priority resource heterogeneous encoder configuration model, based on a Transformer multi-level design and a greedy degradation algorithm, prioritizing the denoising quality of high-priority areas under limited computational power constraints. This achieves efficient cross-platform adaptation from high-end servers to low-end mobile terminals, striking a balance between interpretability and robustness. Compared to existing technologies, this method significantly improves denoising quality and computational resource utilization efficiency. Attached Figure Description
[0009] Figure 1 Flowchart of a method for denoising light field images. Detailed Implementation
[0010] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.
[0011] Reference Figure 1 As shown, the light field image denoising method includes: S1. Based on several pre-set image databases, obtain several original image light field data, establish a local semantic scene classification model, analyze the local semantics of each image light field, output the scene type to which the local semantics of each image light field belongs, and obtain the local scene heat map of each image light field. Step S1 specifically includes: Based on several pre-set image databases, several original image light field data are obtained. According to the sub-aperture image extraction, the central view and the neighboring view of the image light field are divided for early fusion and stacking. The pixel values of the divided images are normalized to obtain several image light field tensors. Based on several pre-set image databases, the LabelMe pixel annotation tool is used to create a central viewpoint label map according to the scene type of the center viewpoint pixels marked by manual annotation. The disparity map between the domain viewpoint and the central viewpoint label map is calculated to perform light field angle consistency mapping to obtain the domain viewpoint label map. Several image light field tensors are used as feature data, and central viewpoint label maps and domain viewpoint label maps are used as label data to construct the original training dataset for scene types. Further, scene types include, but are not limited to, N types such as: portraits, skin regions, text symbols, sky, grass, trees, buildings, roads, vehicles, and water bodies. M original light field images are acquired, and pixel-level semantic annotation tools (such as LabelMe) are used to manually annotate the central viewpoint of each image, generating corresponding label maps. The value of each pixel represents its type index (0 to N-1). Utilizing the parallax information inherent in the light field data, a parallax propagation algorithm is used to automatically map the central viewpoint label map onto the remaining sub-aperture images, generating semantic label data for all viewpoints.
[0012] Based on DeepLabV3 semantic image segmentation encoder-decoder, a local semantic scene classification model is established. The original training dataset of scene types is used as input. The encoder convolutional block progressively downsamples several image light field tensors to extract local low-order object textures and local high-order object semantics of the image light field, and constructs a local multi-scale feature map of the image light field. By utilizing ASPP (Space Pyramid Pooling with Dilated Space) and employing multiple dilated convolutions with different dilation rates, the context information of local high-order semantic feature maps in the local multi-scale feature maps of the image light field is processed in parallel and then concatenated. The input two-dimensional convolution is used to progressively transpose the current local high-order semantic feature map, perform convolution upsampling, and concatenate the channel dimension of the output skip connection features of the convolution block to obtain the context-enhanced feature map of the image light field. Based on the decoder convolutional block, several output channels are established according to the total number of scene types. The probability vector distribution of each pixel position belonging to each scene type in the context enhancement feature map of the image light field is verified by the Softmax function. The scene type corresponding to the maximum probability of each pixel position belonging to each scene type is used as the hard label of the pixel, and the probability vector of each pixel position is used as the soft label to obtain the local semantic classification mask of each image light field. As a further explanation, the local semantic classification mask of the image light field is a two-dimensional matrix, and the value of each matrix element is the semantic scene category index of the corresponding pixel; Create a blank matrix of the same size as the original image, traverse the semantic scene type label of each pixel position in the local semantic classification mask of the light field of each image, and assign it to the blank matrix of the same size as the original image with the same coordinates to obtain the local scene heatmap of the light field of each image.
[0013] As a further development, an implementation example of step S1 is provided: S1-11: Extract the sub-aperture image array of the original light field data, select the center view image I_center and several neighboring view images {I_nbr1, I_nbr2, ..., I_nbrK}, where K is the number of selected neighboring view images; S1-12: The central viewpoint image and the neighboring viewpoint image are fused and stacked in the channel dimension to form a multi-channel input tensor I_multi=concat(I_center, I_nbr1, ..., I_nbrK), with tensor dimension of H×W×(C×(K+1)), where H and W are the image height and width, and C is the number of color channels of a single-channel image; S1-13: Perform pixel value normalization processing on the multi-channel input tensor I_multi, and linearly map the pixel value range to the interval [0, 1] or [-1, 1] to obtain the normalized image light field tensor I_norm.
[0014] S1-21: Collect M original light field images, and manually annotate the center view of each image pixel by pixel using a pixel-level semantic annotation tool. The annotation types cover N preset scene types, including portraits, skin regions, text symbols, sky, grass, trees, buildings, roads, vehicles, and water bodies, and generate a center view label map L_center, where each pixel value is the corresponding type index from 0 to N-1. S1-22: Calculate the disparity map Disp_k between the central viewpoint and each neighboring viewpoint using the raw light field data. The disparity map is obtained by phase correlation algorithm or optical flow estimation algorithm. S1-23: For the k-th neighboring view, project the label value of each pixel (x, y) in the central view label map L_center to the neighboring view coordinates (x', y')=(x+du·Disp_k(x, y), y+dv·Disp_k(x, y)) according to the disparity map Disp_k, where (du, dv) is the baseline offset of the neighboring view relative to the central view. S1-24: Nearest neighbor interpolation is used for the sub-pixel positions generated during the projection process. When multiple pixels are projected to the same coordinate, the majority voting method is used to determine the final label. The empty areas not covered by the projection are filled with adjacent labels to generate the label map L_nbr_k of the k-th neighborhood view. S1-25: The central viewpoint label map L_center and all neighboring viewpoint label maps {L_nbr_1, ..., L_nbr_K} are used together as label data and paired with the image light field tensor I_norm generated in S1-13 to construct the original training dataset for scene types.
[0015] S1-31: Construct an encoder-decoder semantic segmentation network based on the DeepLabV3 architecture, wherein the encoder uses a residual convolutional neural network as the backbone network; S1-32: The image light field tensor I_norm is input into the encoder and downsampled step by step through multiple convolutional blocks. Each convolutional block contains a convolutional layer, a batch normalization layer, and a ReLU activation function. The downsampling ratios are 2x, 4x, 8x, and 16x, respectively, and output the first scale feature map F1 (1 / 2 size), the second scale feature map F2 (1 / 4 size), the third scale feature map F3 (1 / 8 size), and the fourth scale feature map F4 (1 / 16 size). S1-33: Wherein the first scale feature map F1 encodes local low-order object texture information of the image light field, the fourth scale feature map F4 encodes local high-order object semantic information of the image light field, and the second scale feature map F2 and the third scale feature map F3 encode intermediate-level features between texture and semantics.
[0016] S1-41: Input the fourth-scale feature map F4 output by the encoder into the ASPP hole spatial pyramid pooling module, which contains five parallel branches: First branch: 1×1 convolution, with an expansion rate of 1; Second branch: 3×3 convolution, with an expansion rate of 6; Third branch: 3×3 convolution, with an expansion rate of 12; Fourth branch: 3×3 convolution, with an expansion rate of 18; Fifth branch: Global average pooling followed by a 1×1 convolution; S1-42: The output feature maps of the five parallel branches are concatenated along the channel dimension to obtain the multi-scale context aggregation feature map F_asp; S1-43: Perform channel dimensionality reduction on the multi-scale context aggregation feature map F_asp through 1×1 convolution, and output the dimensionality-reduced high-order semantic feature map F4_ctx; S1-44: The high-order semantic feature map F4_ctx is progressively transposed and upsampled by convolution, with upsampling ratios of 2x, 4x, and 8x. After each upsampling, it is concatenated with the third-scale feature map F3, the second-scale feature map F2, and the first-scale feature map F1 output by the corresponding layer of the encoder along the channel dimension. Then, the features are fused and refined through convolutional blocks, and finally, a context-enhanced feature map F_final with the same size as the original image is output.
[0017] S1-51: Set a 1×1 convolutional layer in the last layer of the decoder, and map the number of channels of the context enhancement feature map F_final to the total number of preset scene types N, to obtain a type score map S with dimensions H×W×N; S1-52: Apply the Softmax function to the N-dimensional vector of each pixel position (i, j) in the type score map S to obtain the probability vector P(i, j) = [p0, p1, ..., p_{N-1}], Σp_c = 1; S1-53: For each pixel position (i, j), take the type index corresponding to the maximum value in the probability vector P(i, j) as the hard label L_hard(i, j) = argmax_cp_c for that pixel, and at the same time retain the probability vector P(i, j) as the soft label. S1-54: Using the hard label L_hard(i,j) as elements, construct a two-dimensional matrix M_seg of size H×W as a local semantic classification mask for the light field of each image, where the value of each matrix element is the semantic scene category index of the corresponding pixel from 0 to N-1. S1-55: Initialize a blank matrix H_local with the same size as the original image and dimensions H×W; S1-56: Traverse each pixel position (i, j) in the local semantic classification mask M_seg, read the semantic scene category label L_hard(i, j) at that position, and assign it to the element with the same coordinate (i, j) in the blank matrix H_local; S1-57: Output the matrix H_local after the assignment is completed as a local scene heatmap of the light field of each image.
[0018] When using it, please refer to the steps outlined above: As a further step, step S1 enhances the robustness of the segmentation network under complex lighting and occlusion conditions through multi-view early fusion, avoids the problem of "semantic confusion caused by insufficient receptive field" by using ASPP multi-scale parallel convolution, preserves clear object boundaries while maintaining classification accuracy by using encoder-decoder skip-layer connections, and ensures the consistency of light field angles by generating full-view semantic labels through the parallax propagation algorithm to avoid subsequent denoising artifacts. At the same time, the soft label output provides uncertainty quantification basis for downstream tasks, thereby achieving high-precision and high-consistency end-to-end mapping from raw light field data to pixel-level semantic heatmaps.
[0019] S2. Establish a prior knowledge database of known scene types, construct a local decision network for global scene types, divide the local scene heatmaps of each image light field, and assign attention priority to the local semantic scene category of each image light field under the global scene type according to the local semantic-global scene attention mechanism. Step S2 specifically includes: Based on several pre-defined image databases, using pre-labeled image data of known scene types, and according to maximum likelihood estimation, the local semantic scene category conditional probability, semantic co-occurrence conditional probability, and spatial layout conditional probability in the image data of known scene types are verified to obtain the local conditional probability distribution in the image data of known scene types and to establish a prior knowledge database of known scene types. As a further development, the local semantic scene category conditional probability refers to: the pixel proportion of each semantic scene category in the image; the semantic co-occurrence conditional probability refers to: the number of spatially adjacent pixel pairs of any two semantic scene categories; the spatial layout conditional probability refers to: the distribution of semantic scene categories within the evenly divided grid. Based on the local scene heatmaps of each image light field, we statistically analyze the histogram of the percentage of pixels of semantic scene categories in the local scene heatmap, the number of semantic scene category pairs of adjacent pixels, and the percentage of each type within the local space, and construct a global feature vector set for each image light field. Based on RandomForest, a decision tree for each scene type is constructed. The global feature vector set of the light field of each image is used as the root node, the prior knowledge of the known scene type is used as the branch node, and the probability distribution of each scene type is used as the leaf node. The maximum decrease of Gini impurity is used as the splitting target to build a local decision network for the global scene type. Based on the global scene type local decision network, according to the Bayesian posterior, read the scene type probability distribution stored in all leaf nodes, and observe the posterior probability of each scene type given the global feature vector of the light field of each image. Based on the local decision network of global scene type, the probability distribution of scene type stored in each leaf node is read as a conditional probability estimate of observing each scene type under a given global feature vector. The output probabilities of all decision trees are arithmetically averaged to obtain the posterior probability estimate of the global scene type pointing to the global feature vector of each image light field. The global scene type of each image light field is generated according to the maximum value of the posterior probability of the global scene type. Step S2 also includes: Based on the cross-attention mechanism, a priority assignment function for the local semantic-global scene attention mechanism is constructed. The global scene type and local semantic scene category of each image light field are mapped to high-dimensional embedding vectors. According to the attention mechanism, the global scene type embedding vector is used as the query and the local semantic scene category embedding vector is used as the key to calculate the attention score between each local semantic scene category and the global scene type. As a further development, since the query embedding vector represents the "attention requirement" of the global scene for the local semantics, and the key embedding vector represents the "attention feature" of the local semantic scene category, and the calculated attention weight reflects the importance of each local semantic in the current scene, only the attention weight (priority) is needed, and there is no need to weight the output to obtain the value. Normalization is performed on the conditional probability estimates of each scene type observed under a given global feature vector. The prior importance weights of each semantic scene category under the global scene type are then weighted and fused with the attention scores between each local semantic scene category and the global scene type to obtain the local semantic scene category attention priority scores for each image light field under the global scene type. Specifically: , in, Prioritize the c-th local semantic scene category for each image light field under the global scene type. The attention score between the c-th local semantic scene category and the global scene type. The prior importance weight of the c-th semantic scene category in the global scene type. This refers to the attention fusion coefficient. As a further development, the attention fusion coefficient is set to 0.7, meaning that 70% depends on attention and 30% depends on prior information. However, when the quality of prior data is high, the initial value of the attention fusion coefficient can be reduced. The specific setting value can be set by the implementation technicians themselves, and will not be described in detail here. As a further development, an implementation example of step S2 is provided: S2-101: Collect a preset number of image data labeled with global scene types, including L types such as family photos, travel photos, landscape photos, street scene photos, and document photos; S2-102: For the labeled image data, the following conditional probability distribution is calculated using the maximum likelihood estimation method: Local semantic scene category conditional probability P(c): The percentage of pixels in the image for each semantic scene category c is calculated using the formula P(c) = count(c) / (H × W), where count(c) is the total number of pixels in category c, and H and W are the image height and width. Semantic co-occurrence conditional probability P(c_i,c_j): Count the number of spatially adjacent (pixel distance less than a preset threshold) pixel pairs of any two semantic scene categories c_i and c_j, and obtain the co-occurrence probability after normalization; Spatial layout conditional probability P(c|grid_k): Divide the image into a uniform 3×3 grid and count the proportion of each semantic scene category within each grid block grid_k; S2-103: Store the conditional probability distribution obtained from the above statistics into the prior knowledge database to form a scenario type prior knowledge base PriorDB={P(c),P(c_i,c_j),P(c|grid_k)|forallc,c_i,c_j,grid_k}.
[0020] S2-201: Input S1 output local scene heatmap H_local, which has a size of H×W, and each element takes a semantic scene category index from 0 to N-1; S2-202: Calculate the pixel percentage of each semantic scene category c in the local scene heatmap, and generate an N-dimensional semantic scene category statistical histogram feature vector V_stats=[freq(0),freq(1),...,freq(N-1)]; S2-203: Construct an N×N semantic co-occurrence matrix Cooccur, where the matrix element Cooccur(i,j) represents the number of spatially adjacent pixel pairs between category i and category j. Flatten the co-occurrence matrix into N... 2 3D eigenvector V_cooccur; S2-204: Divide the local scene heatmap into a 3×3 grid, calculate the proportion of each category in each grid block, and generate a 9×N-dimensional spatial layout feature vector V_layout. S2-205: Concatenate the semantic scene category statistical histogram feature vector V_stats, the semantic co-occurrence feature vector V_cooccur, and the spatial layout feature vector V_layout to form a global feature vector set X_global={V_stats,V_cooccur,V_layout}.
[0021] S2-301: Construct a decision network based on RandomForest, wherein the random forest consists of T decision trees, and the value of T ranges from 50 to 200; S2-302: For each decision tree, the random subspace method is used to randomly select some feature dimensions from the global feature vector set X_global, and to randomly sample some samples from the training dataset for training; S2-303: The global feature vector set of the light field of each image is used as the root node, the prior knowledge of the known scene type is used as the splitting condition of the branch node, and the Gini impurity is used as the evaluation index for node splitting. S2-304: For each internal node, traverse all candidate feature dimensions and candidate thresholds, and select the splitting condition f(x)>threshold that maximizes the decrease in Gini impurity, where f(x) is the value of a certain feature dimension. S2-305: Recursively generate child nodes until the preset maximum depth (range 5 to 15) is reached or the number of samples in the node is lower than the preset minimum sample threshold (range 2 to 10), then mark the current node as a leaf node; S2-306: Each leaf node stores a scene type probability distribution P(G|leaf), representing the proportion of each global scene type in the training samples that reach that leaf node.
[0022] S2-401: Input the global feature vector set X_global of the image to be determined into each decision tree in the random forest; S2-402: For a single decision tree, starting from the root node, determine whether to enter the left child node or the right child node based on the splitting condition of the current node. Repeat this process until the leaf node is reached, and read the scene type probability distribution P_tree(G|X) stored in the leaf node. S2-403: Calculate the posterior probability estimate by taking the arithmetic mean of the output probability distributions of all T decision trees. P(G|X)=(1 / T)×Σ_{t=1}^{T}P_tree_t(G|X); S2-404: Take the global scene type corresponding to the maximum posterior probability as the judgment result: G_pred=argmax_GP(G|X).
[0023] S2-501: Construct a learnable embedding layer that maps the global scene type G of each image's light field to a D-dimensional global embedding vector E_global∈ ^D, and simultaneously map each local semantic scene category c to a local embedding vector E_local(c)∈ of the same dimension. ^D, where the value of D ranges from 64 to 256; S2-502: Employs a cross-attention mechanism, using the global embedding vector E_global as the query and the local embedding vector E_local(c) as the key, to calculate the attention score between each local semantic scene category and the global scene type: Score(c)=(E_global·W·E_local(c)) / √D Where W∈ ^{D×D} is the learnable weight matrix, and · denotes matrix multiplication; S2-503: Normalize the attention scores of all N local semantic scene categories using the Softmax function to obtain the attention weight distribution: Attention(c)=exp(Score(c)) / Σ_{j=1}^{N}exp(Score(j)); S2-504: Take the attention weight Attention(c) as the base priority, multiply it by the priority scaling factor of 100, and obtain the attention priority score: S2-601: Query the Prior Knowledge Database PriorDB constructed in step S2-103, and obtain the Prior Importance Weight Prior(c) of each semantic scene category c under the determined global scene type G. S2-602: The prior importance weight Prior(c) is obtained by reading the conditional probability P(c|G) of each semantic scene category under the global scene type from the prior knowledge database, and normalizing the conditional probability as the prior importance weight. S2-603: The attention priority score Priority_attention(c) calculated in step S2-504 is weighted and fused with the prior importance weight Prior(c). The calculation formula is as follows: Final_Priority(c)=β×Priority_attention(c)+(1-β)×Prior(c)×100 Where β is the attention fusion coefficient, with a value ranging from 0.5 to 0.9; S2-604: When the scene determination confidence score conf_scene output in step S2-405 is lower than the preset threshold, the value of the attention fusion coefficient β is automatically reduced to reduce the dependence on the attention mechanism and increase the proportion of prior knowledge weight.
[0024] When using it, please refer to the steps outlined above: As a further step, step S2 automatically determines the global scene type of the image through a random forest decision network and integrates prior knowledge statistical modeling and a cross-attention mechanism to achieve scene-adaptive local semantic priority assignment. This allows the allocation of denoising resources to be dynamically adjusted according to different scene types such as family photos, travel photos, and landscape photos. At the same time, a balance is achieved between intelligence and robustness through a weighted fusion strategy (70% attention, 30% prior knowledge) and confidence-adaptive fusion coefficient adjustment, avoiding the unreliability of a single mechanism. In addition, by extracting three types of global features—semantic statistical histogram, co-occurrence matrix, and spatial layout—and combining them with the strong generalization ability of random forest ensemble learning, the accuracy and stability of scene determination are significantly improved. Meanwhile, the readability of the prior knowledge base and the traceability of the decision path provide good interpretability for the entire priority assignment process.
[0025] S3. Based on the transformer architecture, establish light field image denoising encoders for several scene types, build a priority resource heterogeneous encoder configuration model, and dynamically divide the local semantic scene category priority of each image light field under the global scene type according to the local semantic scene category priority of each image light field under the global scene type, and generate heterogeneous denoising tasks for the local semantic scene category of each image light field under the global scene type. Step S3 specifically includes: Based on the transformer architecture, according to the local semantic scene categories of each image light field under the global scene type, a multi-layer stacked Block structure light field image denoising encoder of several scene types is established. Several configuration processing levels are preset. The priority of the local semantic scene category of each image light field under the global scene type is used as the configuration processing level selection condition. The real-time callable computing power resources of the device hardware are used as the configuration processing level selection constraint boundary to build a priority resource heterogeneous encoder configuration model. Several processing levels are preset, and the local semantic scene category priority of each image light field under the global scene type is used as the selection condition for the processing level. The specific execution process is as follows: Several processing levels are preset: L0 is the lowest power consumption level, which uses 2 layers of Block, 2 attention heads, 64-dimensional embedding, 2x feedforward expansion factor and 0.1 drop rate, with about 0.3M parameters and 0.5GFLOPs of computation per kilopixel, which is used as the baseline computation level. L1 is the base fast shifter, employing 3 layers of Block, 3 attention heads, 96-dimensional embedding, 2.5x feedforward expansion factor, and a dropout rate of 0.1; L2 is an enhanced quality level, employing 6-layer Block, 6 attention heads, 192-dimensional embedding, 3x feedforward expansion factor, and a dropout rate of 0.15; L3 is the highest fidelity setting, employing 12 layers of blocks, 12 attention heads, 384-dimensional embedding, a 4x feedforward expansion factor, and a dropout rate of 0.2.
[0026] Define the encoder gear set as {L_min, L_min+1, ..., L_max}, where L_min is the index of the lowest gear and L_max is the index of the highest gear. The basic mapping rule is: for every 20 points increase in priority, the gear level increases by 1; priority of 0 points corresponds to the lowest gear L_min, and priority of 100 points corresponds to the highest gear L_max.
[0027] Furthermore, the gear position boundary constraint rules are set: for all semantic scene categories, the value range of the gear position index is restricted to between L_min and L_max.
[0028] Furthermore, a gear upper bound rule is set specifically for semantic scene categories: for preset low-perceived importance semantic scene categories, including the sky category and the grass category, their gear index must not exceed the preset upper bound threshold U(c), where U(c) is less than L_max.
[0029] Furthermore, a lower bound rule for the gear position is set specifically for semantic scene categories: for the preset high-perception-importance semantic scene categories, including: portrait category and skin category, their gear position index shall not be lower than the preset lower bound threshold L(c), where L(c) is greater than L_min.
[0030] Furthermore, a sensitivity coefficient s(c) is assigned to each local semantic scene category c, and the assignment rules are as follows: For semantic scene categories with high perceived importance, including the portrait category and the skin category, the sensitivity coefficient s(c) is set to 1.5; For semantic scene categories with moderate perceived importance, including landmark and text categories, the sensitivity coefficient s(c) is set to 1.0; For semantic scene categories with low perceived importance, including the sky and grass categories, the sensitivity coefficient s(c) is set to 0.5.
[0031] Based on the local semantic scene categories under the current global scene type, the sensitivity coefficient of each local semantic scene category, and the attention score between the local semantic scene categories and the global scene type, calculate the configuration processing level index under the priority of the local semantic scene categories. : , Here, max is the maximum value operation and min is the minimum value operation.
[0032] Step S3 also includes: Based on the priority-based heterogeneous encoder configuration model, several original image light field data are used as input. According to the greedy degradation algorithm, iteratively verifies whether the total computing power consumption of each local semantic scene category configuration processing level under the real-time available computing power resources of the device hardware exceeds the constraint boundary of the real-time available computing power resources. If so, the processing level of each semantic category configuration is reduced step by step according to the priority in ascending order. If not, a local semantic scene category heterogeneous denoising task for each image light field under the global scene type is generated.
[0033] When using it, combine the steps above: As a further step, step S3 presets four differentiated configuration levels based on the Transformer architecture (L0 to L3, ranging from 2 layers, 2 heads, 64 dimensions to 12 layers, 12 heads, 384 dimensions, covering computational loads from 1 to 48 times the baseline). This is combined with a priority-driven level mapping formula and semantic category-specific sensitivity coefficients (1.5 for high sensitivity, 0.5 for low sensitivity) and upper and lower bound constraints, allowing high-priority, high-sensitivity regions to automatically obtain higher configuration levels. Simultaneously, real-time available computing resources are introduced as a constraint boundary, employing a greedy degradation algorithm to progressively downgrade from the lowest priority category until the computing power limit is met. This prioritizes high-perception regions such as portraits and skin, ensuring the highest fidelity processing, while downgrading low-sensitivity regions such as sky and grass to the lowest power consumption level. This achieves optimal allocation of denoising quality under limited computing power constraints. Furthermore, the multi-level design, combined with a parameter truncation inheritance mechanism, supports millisecond-level dynamic switching, enabling the same model to flexibly adapt to various hardware devices from high-end servers to low-end mobile terminals, demonstrating excellent cross-platform portability.
[0034] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.
Claims
1. A method for denoising light field images, characterized in that, include: S1. Based on several pre-set image databases, obtain several original image light field data, establish a local semantic scene classification model, analyze the local semantics of each image light field, output the scene type to which the local semantics of each image light field belongs, and obtain the local scene heat map of each image light field. S2. Establish a prior knowledge database of known scene types, construct a local decision network for global scene types, divide the local scene heatmaps of each image light field, and assign attention priority to the local semantic scene category of each image light field under the global scene type according to the local semantic-global scene attention mechanism. S3. Based on the transformer architecture, establish light field image denoising encoders for several scene types, build a priority resource heterogeneous encoder configuration model, and dynamically divide the local semantic scene category priority of each image light field under the global scene type according to the real-time available computing power resources of the device hardware to generate heterogeneous denoising tasks for the local semantic scene category of each image light field under the global scene type.
2. The light field image denoising method according to claim 1, characterized in that, Step S1 specifically includes: Based on several pre-set image databases, several original image light field data are obtained. According to the sub-aperture image extraction, the central view and the neighboring view of the image light field are divided for early fusion and stacking. The pixel values of the divided images are normalized to obtain several image light field tensors. Based on several pre-set image databases, the LabelMe pixel annotation tool is used to create a central viewpoint label map according to the scene type of the central viewpoint pixels marked by manual annotation. The disparity map between the domain viewpoint and the central viewpoint label map is calculated to perform light field angle consistency mapping to obtain the domain viewpoint label map. Several image light field tensors are used as feature data, and the center viewpoint label map and the domain viewpoint label map are used as label data to construct the original training dataset for scene types. Based on DeepLabV3 semantic image segmentation encoder-decoder, a local semantic scene classification model is established. The original training dataset for scene types is used as input. The encoder convolutional block progressively downsamples several image light field tensors to extract local low-order object textures and local high-order object semantics of the image light field, and constructs a local multi-scale feature map of the image light field. By utilizing ASPP (Space Pyramid Pooling with Dilated Space) and employing multiple dilated convolutions with different dilation rates, the context information of local high-order semantic feature maps in the local multi-scale feature maps of the image light field is processed in parallel and then concatenated. The input two-dimensional convolution is used to progressively transpose the current local high-order semantic feature map, perform convolution upsampling, and concatenate the channel dimension of the output skip connection features of the convolution block to obtain the context-enhanced feature map of the image light field. Based on the decoder convolutional block, several output channels are established according to the total number of scene types. The probability vector distribution of each pixel position belonging to each scene type in the context enhancement feature map of the image light field is verified by the Softmax function. The scene type corresponding to the maximum probability of each pixel position belonging to each scene type is used as the hard label of the pixel, and the probability vector of each pixel position is used as the soft label to obtain the local semantic classification mask of each image light field. Create a blank matrix of the same size as the original image, traverse the semantic scene type label of each pixel position in the local semantic classification mask of the light field of each image, and assign it to the blank matrix of the same size as the original image with the same coordinates to obtain the local scene heatmap of the light field of each image.
3. The light field image denoising method according to claim 2, characterized in that, Step S2 specifically includes: Based on several pre-defined image databases, using pre-labeled image data of known scene types, and according to maximum likelihood estimation, the local semantic scene category conditional probability, semantic co-occurrence conditional probability, and spatial layout conditional probability in the image data of known scene types are verified to obtain the local conditional probability distribution in the image data of known scene types and to establish a prior knowledge database of known scene types. Based on the local scene heatmaps of each image light field, we statistically analyze the histogram of the percentage of pixels of semantic scene categories in the local scene heatmap, the number of semantic scene category pairs of adjacent pixels, and the percentage of each type within the local space, and construct a global feature vector set for each image light field. Based on RandomForest, a decision tree for each scene type is constructed. The global feature vector set of the light field of each image is used as the root node, the prior knowledge of the known scene type is used as the branch node, and the probability distribution of each scene type is used as the leaf node. The maximum decrease of Gini impurity is used as the splitting target to build a local decision network for the global scene type. Based on the global scene type local decision network, according to the Bayesian posterior, read the scene type probability distribution stored in all leaf nodes, and observe the posterior probability of each scene type given the global feature vector of the light field of each image. Based on the local decision network of global scene type, the probability distribution of scene type stored in each leaf node is read as a conditional probability estimate of observing each scene type under a given global feature vector. The output probabilities of all decision trees are arithmetically averaged to obtain the posterior probability estimate of the global scene type pointing to the global feature vector of each image light field. The global scene type of each image light field is generated according to the maximum value of the posterior probability of the global scene type.
4. The light field image denoising method according to claim 3, characterized in that, Step S2 also includes: Based on the cross-attention mechanism, a priority assignment function for the local semantic-global scene attention mechanism is constructed. The global scene type and local semantic scene category of each image light field are mapped to high-dimensional embedding vectors. According to the attention mechanism, the global scene type embedding vector is used as the query and the local semantic scene category embedding vector is used as the key to calculate the attention score between each local semantic scene category and the global scene type. Normalization is performed on the conditional probability estimates of each scene type observed under the given global feature vector. The prior importance weights of each semantic scene category of the global scene type are weighted and fused with the attention scores between each local semantic scene category and the global scene type to obtain the attention priority scores of the local semantic scene categories of each image light field under the global scene type.
5. The light field image denoising method according to claim 4, characterized in that, Step S3 specifically includes: Based on the transformer architecture, a multi-layer stacked Block structured light field image denoising encoder is established according to the local semantic scene categories of each image light field under the global scene type. Several configuration processing levels are preset. The priority of the local semantic scene category of each image light field under the global scene type is used as the configuration processing level selection condition, and the real-time callable computing power resources of the device hardware are used as the configuration processing level selection constraint boundary to build a priority resource heterogeneous encoder configuration model.
6. The light field image denoising method according to claim 5, characterized in that, Step S3 also includes: Based on the priority-based heterogeneous encoder configuration model, several original image light field data are used as input. According to the greedy degradation algorithm, iteratively verifies whether the total computing power consumption of each local semantic scene category configuration processing level under the real-time available computing power resources of the device hardware exceeds the constraint boundary of the real-time available computing power resources. If so, the processing level of each semantic category configuration is reduced step by step according to the priority in ascending order. If not, a local semantic scene category heterogeneous denoising task for each image light field under the global scene type is generated.