A Method for Blind Light Field Image Quality Assessment Based on Dynamic Expert Selection

Abstract

The present invention discloses a method for blind light field image quality assessment based on dynamic expert selection, comprising: generating N groups of cross-coded grayscale images from the sub-aperture images of the light field; extracting features in the horizontal and vertical directions respectively for the N groups of cross-coded grayscale images to obtain horizontal features and vertical features; performing local feature aggregation on the horizontal features and vertical features to obtain several cross-combined features; splicing the several cross-combined features and then performing global feature aggregation to obtain aggregated features; combining the aggregated features with dynamic expert selection, dynamically selecting the two most relevant experts according to probability modeling, and obtaining the quality assessment score of the light field image through the combination of the selected experts, thus completing the quality scoring. The present invention improves the accuracy of blind light field image quality assessment.

Description

Technical Field

[0001] The present invention belongs to the technical field of light field image quality evaluation, and particularly relates to a method for blind light field image quality assessment based on dynamic expert selection. Background Art

[0002] A light field image is an advanced visual data form that records information such as the direction, intensity, and position of light in a three-dimensional scene. Compared with traditional two-dimensional images, light field images can capture multi-view information, including the depth, geometric structure, and reflection characteristics of the scene, and thus have important application values in fields such as virtual reality (VR), augmented reality (AR), computational photography, and medical imaging. However, its complex structure and high-dimensional data characteristics pose many challenges to quality assessment.

[0003] The core problems faced by light field image quality assessment include high dimensionality and information redundancy, complex interactions of multi-view information, and scarcity of labeled data. A light field image is composed of multiple sub-aperture images (SAIs), and there may be quality differences when these images record scene information. However, not all view information contributes equally to the overall quality judgment. In addition, the multi-view information of light field images has high correlation and geometric consistency, but traditional quality assessment methods often ignore this internal relationship and only focus on the analysis of a single view or simple features, making it difficult to comprehensively reflect the overall quality of the light field. At the same time, light field image quality assessment, as a problem of no-reference scene quality assessment, faces the challenge of scarce labeled data, making it a key research point on how to achieve efficient assessment with limited labeled data.

[0004] In recent years, deep learning methods have been widely applied to the field of light field image quality assessment. Models such as convolutional neural networks (CNNs) and vision transformers (ViTs) can automatically extract and analyze light field image features and have made certain progress. However, these methods have limitations in processing the multi-view information of light field images, failing to fully capture the geometric structure and multi-view correlation features between views, resulting in insufficient adaptability to complex distortion types. In addition, existing methods mostly use fixed fully connected layers and lack the ability to dynamically adjust weight allocation, which limits the generalization ability and actual effect of the model in different scenarios and distortion types. Summary of the Invention

[0005] To solve the above technical problems, the present invention proposes a method for blind light field image quality assessment based on dynamic expert selection, which improves the accuracy of blind light field image quality assessment.

[0006] To achieve the above object, the present invention provides a method for blind light field image quality assessment based on dynamic expert selection, including:

[0007] Generate N groups of cross-coded grayscale images from the sub-aperture images of the light field;

[0008] Extract features of the N groups of cross-coded grayscale images in the horizontal and vertical directions respectively to obtain horizontal features and vertical features;

[0009] Aggregate the horizontal features and vertical features locally to obtain several cross-combined features;

[0010] Stitch several cross-combined features and then perform global feature aggregation to obtain aggregated features;

[0011] Combine the aggregated features with dynamic expert selection, dynamically select the two most relevant experts according to probability modeling, and obtain the quality evaluation score of the light field image through the combination of the selected experts to complete the quality scoring.

[0012] Optionally, the array specification of the sub-aperture images of the light field is n×n.

[0013] Optionally, generating N groups of cross-coded grayscale images from the sub-aperture images of the light field includes:

[0014] Select target images from the light field sub-aperture image array in different diagonal directions, where each diagonal direction includes multiple horizontal and vertical cross-combinations to form a cross-coded image combination, and each cross-coded image combination is composed of the same number of horizontal sub-aperture images and vertical sub-aperture images;

[0015] Superimpose the horizontal sub-aperture images in the horizontal direction and superimpose the vertical sub-aperture images in the vertical direction to generate N groups of cross-coded grayscale images.

[0016] Optionally, the different diagonal directions are the left diagonal and the right diagonal respectively.

[0017] Optionally, obtaining horizontal features and vertical features includes:

[0018] Using the sliding window mechanism and the self-attention mechanism, extract multi-scale features layer by layer from the cross-coded grayscale images, capture the detail changes in the local area of the light field image and the geometric correlation between different perspectives, and obtain horizontal features and vertical features.

[0019] Optionally, obtaining several cross-combined features includes:

[0020] Perform multi-scale decomposition on the horizontal features and vertical features respectively, and extract local geometric information at different spatial scales;

[0021] Fuse local geometric information at different scales layer by layer, capture the local geometric distortion patterns in the multi-view information of the light field image, and obtain several cross-combined features.

[0022] Optionally, obtaining the aggregated features includes:

[0023] Perform spatial attention mechanism processing on the stitched feature maps of several cross-combined features, and obtain the key regions and global structure information in the image by calculating the importance weights of each spatial position in the feature maps;

[0024] Perform channel attention mechanism processing on the stitched feature maps of several cross-combined features, highlight the important channel information in the feature maps by calculating the importance weights of each channel, suppress the redundant channel information, and obtain the aggregated features.

[0025] Optionally, obtaining the quality assessment score of the light field image includes:

[0026] Calculate the selection probabilities of each expert for the aggregated features through a gating network, and select the two experts with the highest weights;

[0027] Each expert focuses on different quality dimensions of the light field image, and the outputs of the two experts are weighted and fused to obtain the quality assessment score of the light field image.

[0028] Technical effects of the present invention: The present invention discloses a method for blind light field image quality assessment based on dynamic expert selection. By introducing cross-coding representation, multi-view feature fusion, and dynamic expert selection mechanism, it can effectively capture the local geometric distortion and global feature correlation of the light field image, and greatly improve the adaptability to complex distortion types. At the same time, the dynamic expert selection module flexibly adjusts the evaluation strategy according to the image characteristics, significantly improving the generalization performance and evaluation accuracy of the model under multi-scene and multi-distortion conditions, and providing an efficient and reliable solution for light field image quality assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings constituting a part of this application are used to provide a further understanding of this application. The schematic embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation to this application. In the drawings:

[0030] Figure 1 is a schematic flowchart of a method for blind light field image quality assessment based on dynamic expert selection according to an embodiment of the present invention;

[0031] Figure 2 is a framework diagram of the overall detailed process according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0033] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0034] As Figure 1 - Figure 2 shown, in this embodiment, a method for blind light field image quality assessment based on dynamic expert selection is provided, including:

[0035] Generating N groups of cross-coded grayscale images from the light field sub-aperture images;

[0036] Performing feature extraction on the N groups of cross-coded grayscale images in the horizontal and vertical directions respectively to obtain horizontal features and vertical features;

[0037] Performing local feature aggregation on the horizontal features and vertical features to obtain several cross-combined features;

[0038] After splicing several cross-combined features, performing global feature aggregation to obtain aggregated features;

[0039] Combining the aggregated features with dynamic expert selection, dynamically selecting the two most relevant experts according to probability modeling, and obtaining the quality assessment score of the light field image through the combination of the selected experts to complete the quality scoring.

[0040] Further, the array specification of the light field sub-aperture image is n×n.

[0041] Further, generating N groups of cross-coded grayscale images from the light field sub-aperture images includes:

[0042] Selecting target images from the light field sub-aperture image array in different diagonal directions, where each diagonal direction includes multiple horizontal and vertical cross-combinations to form a cross-coded image combination, and each cross-coded image combination is composed of the same number of horizontal sub-aperture images and vertical sub-aperture images;

[0043] Superimposing the horizontal sub-aperture images in the horizontal direction and superimposing the vertical sub-aperture images in the vertical direction to generate N groups of cross-coded grayscale images.

[0044] Specifically, first, select from the 9×9 sub-aperture image of the light field in two diagonal directions. Each diagonal direction includes multiple horizontal and vertical intersections to form a multi-dimensional perspective input with cross-coding. Specifically, each combination consists of three horizontal sub-aperture images and three vertical sub-aperture images. For each group of combinations, a corresponding grayscale image is generated. During the generation of the grayscale image, the horizontal images are stacked in the horizontal direction, while the vertical images are stacked in the vertical direction. There are a total of eight groups of stacked grayscale images, four horizontal and four vertical, each with a dimension of (batch_size, c, h, w).

[0045] Furthermore, the different diagonal directions are the left diagonal and the right diagonal respectively.

[0046] Furthermore, obtaining the features in the horizontal direction and the vertical direction includes:

[0047] Using the sliding window mechanism and the self-attention mechanism, extract multi-scale features layer by layer from the cross-coded grayscale images, capture the detailed changes in local regions of the light field image and the geometric correlations between different perspectives, and obtain the features in the horizontal direction and the vertical direction.

[0048] Specifically, first input the obtained cross-coded grayscale images into 8 pre-trained Swin Transformer modules to extract features from the images in the horizontal and vertical directions respectively. The cross-coded grayscale images contain the perspective information generated from horizontal and vertical stacking. The Swin Transformer module extracts its multi-scale features layer by layer. The Swin Transformer module can effectively capture the detailed changes in local regions of the light field image and the geometric correlations between different perspectives by using the sliding window mechanism and the self-attention mechanism. By combining multi-head self-attention and hierarchical structures in each module, it ensures the efficiency and robustness of the model in extracting features from multi-perspective information, and finally generates the feature representations corresponding to the horizontal and vertical directions respectively, each with a dimension of (batch_size, c, h, w).

[0049] Furthermore, obtaining the features of several cross combinations includes:

[0050] Perform multi-scale decomposition on the features in the horizontal direction and the vertical direction respectively to extract the local geometric information at different spatial scales;

[0051] Fuse the local geometric information at different scales layer by layer to capture the local geometric distortion patterns in the multi-perspective information of the light field image and obtain the features of several cross combinations.

[0052] Specifically, the horizontal and vertical features output by the Swin Transformer are spliced according to cross - combination and input into the Pyramid SqueezeAttention (PSA) module. Specifically, the PSA module first performs multi - scale decomposition on the feature maps in the horizontal and vertical directions respectively to extract local geometric information at different spatial scales. At each scale, the PSA module highlights the geometric distortion features of the significant regions in the feature map through channel compression, while suppressing the influence of redundant information. Subsequently, the PSA module fuses the features at different scales layer by layer to capture the local geometric distortion patterns in the multi - perspective information of the light field image, and finally outputs four cross - combined features.

[0053] Furthermore, obtaining the aggregated features includes:

[0054] Performing spatial attention mechanism processing on several cross - combined feature maps after splicing, and obtaining the key regions and global structure information in the image by calculating the importance weights of each spatial position in the feature map;

[0055] Performing channel attention mechanism processing on several cross - combined feature maps after splicing, highlighting the important channel information in the feature map by calculating the importance weights of each channel, suppressing the redundant channel information, and obtaining the aggregated features.

[0056] Specifically, after splicing the four cross - combined features, global aggregation is performed through the Spatial and Channel SynergisticAttention (SCSA) module. The SCSA module can enhance the sensitivity of the model to global distortion of the light field while reducing the interference of redundant information. Specifically, the SCSA module first performs spatial attention mechanism processing on the spliced features, and by calculating the importance weights of each spatial position in the feature map, enables the model to pay more attention to the key regions and global structure information in the image. Then, the module performs channel attention mechanism processing on the features, and by calculating the importance weights of each channel, further highlights the important channel information in the feature map while suppressing the redundant channel information, and finally outputs the aggregated features.

[0057] Furthermore, obtaining the quality evaluation score of the light field image includes:

[0058] Calculating the selection probabilities of each expert for the aggregated features through a gating network, and selecting the two experts with the highest weights;

[0059] Each expert focuses on different quality dimensions of the light field image, and the outputs of the two experts are fused through weighting to obtain the quality evaluation score of the light field image.

[0060] Specifically, in the feature output stage, a Mixture of Experts (MoE) module is adopted. According to the analysis of the input aggregated features by the Gating Network, two most relevant expert networks are dynamically selected. Each expert is good at dealing with different types of distortions. The Gating Network calculates the selection probabilities of each expert based on the input aggregated features and selects the two experts with the highest weights to output results. Finally, the outputs of the two experts are fused through weighted summation to generate the regression score.

[0061] A specific application example of the present invention is as follows:

[0062] Multiple groups of perspectives are selected from the 9×9 optical field sub-aperture images as the input for subsequent feature extraction. First, images are selected along two diagonal directions, each direction containing multiple horizontal and vertical cross combinations to form multi-dimensional perspective inputs with cross-coding. Each cross combination consists of three horizontal sub-aperture images and three vertical sub-aperture images. In each cross combination, the images are superimposed to generate grayscale images. The horizontal images are superimposed in the horizontal direction, while the vertical images are superimposed in the vertical direction.

[0063] After the process of generating cross-coding, the horizontal and vertical sub-aperture images are respectively superimposed to obtain 8 groups of cross-coded grayscale images. The dimension of each group of grayscale images is (batch_size, 3, 224, 224), where batch_size is the batch size, 3 indicates that each group contains 3 images (i.e., three horizontal images and three vertical images in a group), and 224 is the spatial resolution of the images.

[0064] The obtained 8 groups of cross-coded grayscale images are input into 8 pre-trained Swin Transformer modules for feature extraction. Swin Transformer is a model that can effectively process local features and multi-scale features of images, and can capture detailed information in images by using the sliding window mechanism and self-attention mechanism.

[0065] Each Swin Transformer module processes the images in the horizontal and vertical directions respectively, and extracts image features layer by layer through multi-level convolution and self-attention mechanisms. This process effectively captures the information of different perspectives in the images and deals with the changes in geometric structures and local regions, ensuring that representative features are extracted from images with different perspectives. The output feature dimension of each Swin Transformer module is (batch_size, 1024, 7, 7), that is, each group of features contains 1024 channels, and the spatial resolution is reduced to 7×7. This process provides rich feature information for subsequent geometric distortion processing.

[0066] The features output by 8 groups of Swin Transformers are concatenated in a cross - combination manner to obtain 4 groups of concatenated features. The dimension of each group of concatenated features is (batch_size, 2048, 7, 7). Here, 2048 is because each cross - combination contains features in both horizontal and vertical directions, and each feature contains 1024 channels. Then, these concatenated features are input into the Pyramid Squeeze Attention (PSA) module for processing. The PSA module processes each group of feature maps through multi - scale decomposition to capture local geometric distortions. Specifically, the PSA module first performs multi - scale decomposition on each group of features, extracts geometric information at different spatial scales, so as to accurately detect subtle geometric distortions in the light field image. To highlight significant regions and suppress redundant information, the PSA module strengthens the representation of important features through the Channel Squeeze mechanism and reduces the interference of unimportant regions. The processed features are returned again, and the dimension of each group of features is (batch_size, 1024, 7, 7), and this feature represents the processed geometric distortion information in the light field image.

[0067] The 4 groups of features processed by PSA are concatenated to obtain global features with a dimension of (batch_size, 4096, 7, 7). The concatenated features contain image information from different perspectives and scales, representing the multi - dimensional features of the light field image. Then, a 1×1 convolutional layer is used to perform channel dimensionality reduction on the concatenated features, reducing the number of channels from 4096 to 1024, and the dimension becomes (batch_size, 1024, 7, 7). This operation helps to reduce redundant information and control the feature dimension within an appropriate range, ensuring that subsequent processing can focus on important features. Then, the dimension - reduced features are input into the Spatial and Channel Synergistic Attention (SCSA) module. The SCSA module combines Spatial Attention and Channel Attention to further process the features. The Spatial Attention mechanism highlights the features of key regions by calculating the importance of each spatial position in the image; the Channel Attention mechanism further emphasizes the channels that have an important impact in the evaluation by calculating the importance of each channel. Finally, the features output by the SCSA module have a dimension of (batch_size, 1024, 7, 7), which indicates that the global feature information of the light field image has been effectively aggregated and redundant information has been suppressed.

[0068] The Mixture of Experts (MoE) optimizes for the complex multi-dimensional characteristics and quality assessment requirements of light field images by introducing a dynamic expert selection mechanism. First, the features output by the SCSA module are reorganized to obtain a feature matrix with the shape of (batch_size, 49, 1024), where 49 represents the number of spatial regions of the light field image, and 1024 is the number of feature channels for each region. Next, the first token of each sample (i.e., the features of the first spatial region) is extracted from this matrix and transformed into the shape of (batch_size, 1024). This step extracts the global feature representation of the light field image and captures the overall information of the image.

[0069] At the core of this step, the MoE module analyzes the input features through a gating network (GatingNetwork) to generate the weight distribution for each expert. According to this distribution, the gating network adaptively selects the two most relevant experts from 8 candidate experts for weighted fusion. The top_k = 2 in the configuration ensures that only the two experts with the highest weights are selected to participate in the output each time, ensuring that the selected experts can most effectively process the key quality information in the light field image. Each expert focuses on different quality dimensions of the light field image, such as sharpness, contrast, or noise characteristics. Under the guidance of the gating network, they can conduct in-depth analysis on the multi-view characteristics and complex geometric distortion patterns of the image. By weighted fusing the outputs of these experts, a quality assessment score for the light field image is finally generated, with the dimension of (batch_size, 1), representing the quality score of the image.

[0070] The present invention proposes a method for dynamic expert selection-based blind light field image quality assessment. First, from the 9×9 light field sub-aperture images, multiple horizontal and vertical cross combinations are selected along two diagonal directions. Each group contains three horizontal images and three vertical images, forming a cross-coded multi-dimensional perspective input, and a grayscale image is generated by superposition. Then, these cross-coded images are input into 8 pre-trained Swin Transformer modules for feature extraction. The extracted multi-scale features are processed by the Pyramid Squeeze Attention (PSA) module to highlight local geometric distortion information and reduce redundancy. Then, the features processed by the PSA module enter the Spatial and Channel Synergistic Attention (SCSA) module through splicing and channel dimension reduction operations to further aggregate global features and optimize the interference of redundant information. Finally, the Mixture of Experts (MoE) module is used to dynamically select the two most relevant experts according to the aggregated features input through the gating network for weighted fusion, and finally output the quality assessment score of the light field image. This method effectively improves the accuracy and robustness of light field image quality assessment, can adapt to multi-perspective and complex distortion scenarios, and has strong practical application value.

[0071] The method for dynamic expert selection-based blind light field image quality assessment of the present invention has broad application prospects and significant advantages. This method breaks through the limitations of traditional assessment techniques in light field image quality assessment, can accurately process multi-perspective, complex distortion types, and high-dimensional data characteristics, and is applicable to fields with high requirements for light field image quality such as virtual reality (VR), augmented reality (AR), computational photography, and medical imaging. Through the cross-coding technology, the geometric relationship of multi-perspective information in the light field image is fully utilized, avoiding the limitations of single-perspective assessment in traditional methods, thereby improving the assessment accuracy and comprehensiveness.

[0072] In addition, through the dynamic expert selection (MoE) mechanism of the present invention, the most relevant expert network is automatically selected according to the input features. This design enables the model to flexibly adjust the assessment strategy according to the specific distortion type and quality dimension of the light field image. Compared with the limitations of fixed weight allocation in traditional methods, the MoE mechanism can effectively optimize the utilization of computing resources while ensuring high assessment accuracy, improving the assessment efficiency. Combined with the Pyramid Squeeze Attention (PSA) and Spatial and Channel Synergistic Attention (SCSA) modules, the present invention can strengthen the model's sensitivity to the details and structure of the light field image when dealing with local and global geometric distortions, effectively reduce the interference of redundant information, thereby enhancing the robustness and generalization ability of the model.

[0073] The above are only the preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for blind light field image quality assessment based on dynamic expert selection, characterized in that Including: Generating N groups of cross - coded grayscale images from the sub - aperture images of the light field; Generating N groups of cross - coded grayscale images from the sub - aperture images of the light field includes: Selecting target images from the light - field sub - aperture image array in different diagonal directions, where each diagonal direction includes multiple horizontal and vertical cross - combinations to form a cross - coded image combination, and each of the cross - coded image combinations is composed of the same number of horizontal sub - aperture images and vertical sub - aperture images; Superposing the horizontal sub - aperture images in the horizontal direction and superposing the vertical sub - aperture images in the vertical direction to generate N groups of cross - coded grayscale images; Performing feature extraction on the N groups of cross - coded grayscale images in the horizontal and vertical directions respectively to obtain the features in the horizontal direction and the features in the vertical direction; Performing local feature aggregation on the features in the horizontal direction and the features in the vertical direction to obtain the features of several cross - combinations; Performing global feature aggregation after splicing the features of several cross - combinations to obtain the aggregated features; Obtaining the aggregated features includes: Performing a spatial attention mechanism process on the spliced feature maps of several cross - combinations, and obtaining the key regions and global structure information in the image by calculating the importance weights of each spatial position in the feature map; Performing a channel attention mechanism process on the spliced feature maps of several cross - combinations, highlighting the important channel information in the feature map by calculating the importance weights of each channel, suppressing the redundant channel information, and obtaining the aggregated features; Combining the aggregated features with dynamic expert selection, dynamically selecting the two most relevant experts according to probability modeling, and obtaining the quality evaluation score of the light - field image through the combination of the selected experts to complete the quality scoring; The dynamic expert selection method includes: using the MoE module to analyze the input features through a gating network to generate the weight distribution of each expert; according to the weight distribution, the gating network adaptively selects the two most relevant experts from 8 candidate experts for weighted fusion; only the two experts with the highest weights are selected to participate in the output each time; each expert focuses on different quality dimensions of the light - field image; finally, generating the quality evaluation score of the light - field image through weighted fusion of the outputs of these experts.

2. The method for blind light field image quality assessment based on dynamic expert selection according to claim 1, wherein The array specification of the light - field sub - aperture images is n×n.

3. The method for blind light field image quality assessment based on dynamic expert selection as claimed in claim 1, wherein The different diagonal directions are the left diagonal and the right diagonal respectively.

4. The method for blind light field image quality assessment based on dynamic expert selection according to claim 1, wherein Obtaining the features in the horizontal direction and the features in the vertical direction includes: Using the sliding window mechanism and the self - attention mechanism to extract multi - scale features layer by layer from the cross - coded grayscale images, capturing the detail changes in the local regions of the light - field image and the geometric correlations between different perspectives, and obtaining the features in the horizontal direction and the features in the vertical direction.

5. The method for blind light field image quality assessment based on dynamic expert selection according to claim 1, wherein Obtaining the features of several cross - combinations includes: Performing multi - scale decomposition on the features in the horizontal direction and the features in the vertical direction respectively, and extracting the local geometric information at different spatial scales; Fusing the local geometric information at different scales layer by layer, capturing the local geometric distortion patterns in the multi - perspective information of the light - field image, and obtaining the features of several cross - combinations.

6. The method for blind light field image quality assessment based on dynamic expert selection according to claim 1, wherein Obtaining the quality evaluation score of the light - field image includes: Calculating the selection probabilities of each expert for the aggregated features through a gating network and selecting the two experts with the highest weights; Each expert focuses on different quality dimensions of the light field image, and the outputs of the two experts are weighted and fused to obtain the quality evaluation score of the light field image.

Images