Blind light field image quality evaluation method based on dynamic expert selection

Abstract

The invention discloses a blind light field image quality evaluation method based on dynamic expert selection. The method comprises the following steps: generating N groups of cross coding grey-scale maps from a light field sub-aperture image; performing feature extraction on the N groups of cross-coded grey-scale maps in the horizontal direction and the vertical direction to obtain features in the horizontal direction and features in the vertical direction; performing local feature aggregation on the features in the horizontal direction and the features in the vertical direction to obtain a plurality of cross-combined features; splicing a plurality of crossed and combined features and then carrying out global feature aggregation to obtain aggregated features; and combining the aggregated features with dynamic expert selection, dynamically selecting two most relevant experts according to probability modeling, and obtaining a quality evaluation score of the light field image through the combination of the selected experts to complete quality scoring. According to the invention, the blind light field image quality evaluation precision is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of light field image quality evaluation, and in particular relates to a method for blind light field image quality evaluation based on dynamic expert selection. Background Art

[0002] Light field images are a form of advanced visual data that records information such as the direction, intensity, and position of light in a three-dimensional scene. Compared with traditional two-dimensional images, light field images can capture multi-perspective information, including the depth, geometric structure, and reflection characteristics of the scene. Therefore, they have important application value in virtual reality (VR), augmented reality (AR), computational photography, and medical imaging. However, their complex structure and high-dimensional data characteristics bring many challenges to quality assessment.

[0003] The core issues facing light field image quality assessment include high dimensionality and information redundancy, complex interactions of multi-view information, and scarcity of annotated data. Light field images are composed of multiple sub-aperture images (SAIs), which may have quality differences when recording scene information, but not all view information contributes equally to the overall quality judgment. In addition, the multi-view information of light field images is highly correlated and geometrically consistent, but traditional quality assessment methods often ignore this inherent correlation and only focus on the analysis of a single view or simple features, which makes it difficult to fully reflect the overall quality of the light field. At the same time, as a no-reference scene quality assessment problem, light field image quality assessment faces the challenge of scarce annotated data, making how to achieve efficient evaluation with limited annotated data the key to research.

[0004] In recent years, deep learning methods have been widely used in the field of light field image quality assessment. Models such as convolutional neural networks (CNN) and visual transformers (ViT) can automatically extract and analyze light field image features, and have made some progress. However, these methods have limitations when processing multi-view information of light field images. They fail to fully capture the geometric structure between viewpoints and multi-view correlation features, resulting in insufficient adaptability to complex distortion types. In addition, existing methods mostly use fixed fully connected layers and lack the ability to dynamically adjust weight distribution, which limits the generalization ability and actual effect of the model in different scenarios and distortion types. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a method for blind light field image quality assessment based on dynamic expert selection, which improves the accuracy of blind light field image quality assessment.

[0006] To achieve the above object, the present invention provides a method for blind light field image quality assessment based on dynamic expert selection, comprising:

[0007] Generate N groups of cross-coded grayscale images from the light field sub-aperture images;

[0008] Extracting features in the horizontal direction and the vertical direction for the N groups of cross-coded grayscale images to obtain features in the horizontal direction and features in the vertical direction;

[0009] Perform local feature aggregation on the horizontal and vertical features to obtain several cross-combination features;

[0010] After several cross-combined features are spliced ​​together, global feature aggregation is performed to obtain aggregated features;

[0011] The aggregated features are combined with dynamic expert selection, and the two most relevant experts are dynamically selected according to probability modeling. The quality assessment score of the light field image is obtained by selecting a combination of experts to complete the quality scoring.

[0012] Optionally, the array size of the light field sub-aperture image is n×n.

[0013] Optionally, generating N groups of cross-coded grayscale images from the light field sub-aperture images includes:

[0014] Selecting target images from the light field sub-aperture image array according to different diagonal directions, wherein each diagonal direction includes a plurality of horizontal and vertical cross combinations to form a cross-coded image combination, and each of the cross-coded image combinations is composed of the same number of horizontal sub-aperture images and vertical sub-aperture images;

[0015] The horizontal sub-aperture images are stacked in the horizontal direction, and the vertical sub-aperture images are stacked in the vertical direction to generate N groups of cross-coded grayscale images.

[0016] Optionally, the different diagonal directions are respectively a left diagonal and a right diagonal.

[0017] Optionally, obtaining the horizontal feature and the vertical feature includes:

[0018] By using a sliding window mechanism and a self-attention mechanism, multi-scale features are extracted layer by layer from the cross-coded grayscale image to capture detail changes in local areas of the light field image and geometric associations between different perspectives, thereby obtaining horizontal and vertical features.

[0019] Optionally, the features of obtaining several cross combinations include:

[0020] Perform multi-scale decomposition on the horizontal and vertical features respectively to extract local geometric information at different spatial scales;

[0021] The local geometric information of different scales is fused layer by layer to capture the local geometric distortion patterns in the multi-view information of the light field image and obtain several cross-combination features.

[0022] Optionally, obtain aggregate features including:

[0023] The spatial attention mechanism is used to process the spliced ​​cross-combined feature maps. The key areas and global structural information in the image are obtained by calculating the importance weight of each spatial position in the feature map.

[0024] The channel attention mechanism is used to process the concatenated cross-combined feature maps. By calculating the importance weight of each channel, the important channel information in the feature map is highlighted, the redundant channel information is suppressed, and the aggregated features are obtained.

[0025] Optionally, obtaining a quality assessment score for a light field image includes:

[0026] Calculate the selection probability of each expert based on the aggregated features through a gating network, and select the two experts with the highest weights;

[0027] Each expert focuses on different quality dimensions of light field images, and the outputs of the two experts are weighted fused to obtain the quality assessment score of the light field image.

[0028] Technical effect of the invention: The invention discloses a method for blind light field image quality assessment based on dynamic expert selection. By introducing cross-coding representation, multi-view feature fusion and dynamic expert selection mechanism, it can effectively capture the local geometric distortion and global feature association of light field images, greatly improving the adaptability to complex distortion types. At the same time, the dynamic expert selection module flexibly adjusts the evaluation strategy according to the image characteristics, significantly improving the generalization performance and evaluation accuracy of the model under multi-scene and multi-distortion conditions, and providing an efficient and reliable solution for light field image quality assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0030] Figure 1 A flowchart of a method for blind light field image quality assessment based on dynamic expert selection according to an embodiment of the present invention;

[0031] Figure 2 It is a framework diagram of the overall detailed process of an embodiment of the present invention. DETAILED DESCRIPTION

[0032] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0033] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0034] like Figure 1-Figure 2 As shown, this embodiment provides a method for blind light field image quality assessment based on dynamic expert selection, including:

[0035] Generate N groups of cross-coded grayscale images from the light field sub-aperture images;

[0036] Extracting features in the horizontal direction and the vertical direction for the N groups of cross-coded grayscale images to obtain features in the horizontal direction and features in the vertical direction;

[0037] Perform local feature aggregation on the horizontal and vertical features to obtain several cross-combination features;

[0038] After several cross-combined features are spliced ​​together, global feature aggregation is performed to obtain aggregated features;

[0039] The aggregated features are combined with dynamic expert selection, and the two most relevant experts are dynamically selected according to probability modeling. The quality assessment score of the light field image is obtained by selecting a combination of experts to complete the quality scoring.

[0040] Furthermore, the array size of the light field sub-aperture image is n×n.

[0041] Further, generating N groups of cross-coded grayscale images from the light field sub-aperture images includes:

[0042] Selecting target images from the light field sub-aperture image array according to different diagonal directions, wherein each diagonal direction includes a plurality of horizontal and vertical cross combinations to form a cross-coded image combination, and each of the cross-coded image combinations is composed of the same number of horizontal sub-aperture images and vertical sub-aperture images;

[0043] The horizontal sub-aperture images are stacked in the horizontal direction, and the vertical sub-aperture images are stacked in the vertical direction to generate N groups of cross-coded grayscale images.

[0044] Specifically, first select from the 9×9 light field sub-aperture image according to two diagonal directions. Each diagonal direction includes multiple horizontal and vertical cross combinations to form a cross-encoded multi-dimensional perspective input. Specifically, each combination consists of three horizontal sub-aperture images and three vertical sub-aperture images. For each group of combinations, a corresponding grayscale image is generated. In the process of generating the grayscale image, the horizontal images are superimposed in the horizontal direction, and the vertical images are superimposed in the vertical direction. There are a total of eight groups of superimposed grayscale images, four horizontal and four vertical, each with a dimension of (batch_size, c, h, w).

[0045] Furthermore, the different diagonal directions are respectively a left diagonal and a right diagonal.

[0046] Furthermore, obtaining the features in the horizontal direction and the features in the vertical direction includes:

[0047] By using a sliding window mechanism and a self-attention mechanism, multi-scale features are extracted layer by layer from the cross-coded grayscale image to capture detail changes in local areas of the light field image and geometric associations between different perspectives, thereby obtaining horizontal and vertical features.

[0048] Specifically, the obtained cross-coded grayscale image is first input into 8 pre-trained Swin Transformer modules to extract features from the horizontal and vertical images respectively. The cross-coded grayscale image contains the perspective information generated from the horizontal and vertical superposition, and its multi-scale features are extracted layer by layer through the Swin Transformer module. The Swin Transformer module uses the sliding window mechanism and self-attention mechanism to effectively capture the detail changes of local areas in the light field image and the geometric associations between different perspectives. By combining multi-head self-attention and hierarchical structure in each module, the model is ensured to be efficient and robust in feature extraction of multi-view information, and finally generates feature representations corresponding to the horizontal and vertical directions, each with a dimension of (batch_size, c, h, w).

[0049] Furthermore, several cross-combination features are obtained, including:

[0050] Perform multi-scale decomposition on the horizontal and vertical features respectively to extract local geometric information at different spatial scales;

[0051] The local geometric information of different scales is fused layer by layer to capture the local geometric distortion patterns in the multi-view information of the light field image and obtain several cross-combination features.

[0052] Specifically, the horizontal and vertical features output by Swin Transformer are spliced ​​and input into the Pyramid SqueezeAttention (PSA) module in a cross-combination. Specifically, the PSA module first performs multi-scale decomposition on the feature maps in the horizontal and vertical directions to extract local geometric information at different spatial scales. At each scale, the PSA module highlights the geometric distortion features of the salient areas in the feature map through channel compression, while suppressing the influence of redundant information. Subsequently, the PSA module fuses features of different scales layer by layer to capture the local geometric distortion patterns in the multi-view information of the light field image, and finally outputs four cross-combined features.

[0053] Furthermore, the aggregated features include:

[0054] The spatial attention mechanism is used to process the spliced ​​cross-combined feature maps. The key areas and global structural information in the image are obtained by calculating the importance weight of each spatial position in the feature map.

[0055] The channel attention mechanism is used to process the concatenated cross-combined feature maps. By calculating the importance weight of each channel, the important channel information in the feature map is highlighted, the redundant channel information is suppressed, and the aggregated features are obtained.

[0056] Specifically, after the four cross-combined features are spliced, they are globally aggregated through the Spatial and Channel SynergisticAttention (SCSA) module. The SCSA module can enhance the model's sensitivity to global distortion of the light field while reducing the interference of redundant information. Specifically, the SCSA module first processes the spliced ​​features with a spatial attention mechanism, and by calculating the importance weight of each spatial position in the feature map, the model can pay more attention to the key areas and global structural information in the image. Then, the module processes the features with a channel attention mechanism, and by calculating the importance weight of each channel, it further highlights the important channel information in the feature map, while suppressing redundant channel information, and finally outputs the aggregated features.

[0057] Furthermore, obtaining the quality assessment score of the light field image includes:

[0058] Calculate the selection probability of each expert based on the aggregated features through a gating network, and select the two experts with the highest weights;

[0059] Each expert focuses on different quality dimensions of light field images, and the outputs of the two experts are weighted fused to obtain the quality assessment score of the light field image.

[0060] Specifically, in the feature output stage, the dynamic expert selection (Mixture of Experts, MoE) module is used to dynamically select the two most relevant expert networks based on the analysis of the input aggregate features by the gating network. Each expert is good at targeting different types of distortion. The gating network calculates the selection probability of each expert based on the input aggregate features and selects the two experts with the highest weights to output the results; finally, the outputs of the two experts are weighted and fused to generate a regression score.

[0061] A specific application example of the present invention is as follows:

[0062] Multiple sets of viewpoints are selected from the 9×9 light field subaperture images as input for subsequent feature extraction. First, images are selected along two diagonal directions, where each direction contains multiple horizontal and vertical cross combinations to form a cross-encoded multi-dimensional viewpoint input. Each cross combination consists of three horizontal subaperture images and three vertical subaperture images. In each cross combination, the images are superimposed to generate a grayscale image. Horizontal images are superimposed in the horizontal direction, while vertical images are superimposed in the vertical direction.

[0063] After the cross-coding process, the horizontal and vertical sub-aperture images are superimposed respectively to obtain 8 groups of cross-coded grayscale images. The dimension of each group of grayscale images is (batch_size, 3, 224, 224), where batch_size is the batch size, 3 means that each group contains 3 images (i.e., a group of three horizontal images and three vertical images), and 224 is the spatial resolution of the image.

[0064] The obtained 8 groups of cross-coded grayscale images are input into 8 pre-trained Swin Transformer modules for feature extraction. Swin Transformer is a model that effectively processes local features and multi-scale features of images, and can capture detailed information in images using sliding window mechanism and self-attention mechanism.

[0065] Each Swin Transformer module processes images in the horizontal and vertical directions respectively, and extracts image features layer by layer through multi-level convolution and self-attention mechanisms. This process effectively captures information from different perspectives in the image and handles changes in geometric structures and local areas to ensure that representative features are extracted from images from different perspectives. The output feature dimension of each Swin Transformer module is (batch_size, 1024, 7, 7), that is, each set of features contains 1024 channels, and the spatial resolution is reduced to 7×7. This process provides rich feature information for subsequent geometric distortion processing.

[0066] The 8 sets of features output by Swin Transformer are spliced ​​in a cross-combination manner to obtain 4 sets of spliced ​​features. The dimension of each spliced ​​feature set is (batch_size, 2048, 7, 7), where 2048 is because each cross-combination contains horizontal and vertical features, and each feature contains 1024 channels. Then, these spliced ​​features are input into the Pyramid Squeeze Attention (PSA) module for processing. The PSA module processes each set of feature maps through multi-scale decomposition to capture local geometric distortion. Specifically, the PSA module first decomposes each set of features into multiple scales and extracts geometric information at different spatial scales, so that subtle geometric distortions in light field images can be accurately detected. In order to highlight salient areas and suppress redundant information, the PSA module uses a channel squeeze mechanism to strengthen the representation of important features and reduce interference in unimportant areas. The processed features are returned again, and the dimension of each feature set is (batch_size, 1024, 7, 7). This feature represents the processed geometric distortion information in the light field image.

[0067] The four sets of features processed by PSA are concatenated to obtain global features with a dimension of (batch_size, 4096, 7, 7). The concatenated features contain image information from different perspectives and scales, representing the multi-dimensional features of the light field image. Next, a 1×1 convolutional layer is used to perform channel dimensionality reduction on the concatenated features, reducing the number of channels from 4096 to 1024, and the dimension becomes (batch_size, 1024, 7, 7). This operation helps to reduce redundant information and control the feature dimension within a suitable range, ensuring that subsequent processing can focus on important features. Then, the reduced-dimensional features are input into the Spatial and Channel Synergistic Attention (SCSA) module. The SCSA module combines spatial attention and channel attention to further process the features. The spatial attention mechanism highlights the features of key areas by calculating the importance of each spatial position in the image; the channel attention mechanism further emphasizes the channels that have an important influence in the evaluation by calculating the importance of each channel. Finally, the feature dimension of the SCSA module output is (batch_size, 1024, 7, 7), which means that the global feature information of the light field image has been effectively aggregated and the redundant information has been suppressed.

[0068] The MoE (Mixture of Experts) model optimizes the complex multidimensional characteristics and quality assessment requirements of light field images by introducing a dynamic expert selection mechanism. First, the features output by the SCSA module are reorganized to obtain a feature matrix of shape (batch_size, 49, 1024), where 49 represents the number of spatial regions of the light field image and 1024 is the number of feature channels for each region. Next, the first token (i.e., the feature of the first spatial region) of each sample is extracted from the matrix and converted to the shape of (batch_size, 1024). This step extracts the global feature representation of the light field image and captures the overall information of the image.

[0069] At the core of this step, the MoE module analyzes the input features through a gating network to generate a weight distribution for each expert. Based on this distribution, the gating network adaptively selects the two most relevant experts from the eight candidate experts for weighted fusion. The top_k=2 in the configuration selects only the two experts with the highest weights to participate in the output each time, ensuring that the selected experts can most effectively process the key quality information in the light field image. Each expert focuses on different quality dimensions of the light field image, such as clarity, contrast or noise characteristics. Under the guidance of the gating network, they can conduct in-depth analysis of the multi-view characteristics and complex geometric distortion patterns of the image. By weighted fusion of the outputs of these experts, the quality assessment score of the light field image is finally generated, with the dimension of (batch_size,1), which represents the quality score of the image.

[0070] The present invention proposes a method for dynamic expert selection blind light field image quality assessment. First, multiple horizontal and vertical cross combinations are selected from 9×9 light field sub-aperture images according to two diagonal directions. Each group contains three horizontal images and three vertical images to form a cross-coded multi-dimensional perspective input, and a grayscale image is generated by superposition. Then, these cross-coded images are input into 8 pre-trained Swin Transformer modules for feature extraction. The extracted multi-scale features are processed by the Pyramid Squeeze Attention (PSA) module to highlight local geometric distortion information and reduce redundancy. Then, the features processed by the PSA module enter the Spatial and Channel Synergistic Attention (SCSA) module through splicing and channel dimensionality reduction operations to further aggregate global features and optimize the interference of redundant information. Finally, the MoE (mixed expert model) module is used to dynamically select the two most relevant experts according to the input aggregated features through the gated network, perform weighted fusion, and finally output the quality assessment score of the light field image. This method effectively improves the accuracy and robustness of light field image quality assessment, can adapt to multi-view and complex distortion scenes, and has strong practical application value.

[0071] The method of blind light field image quality assessment based on dynamic expert selection has broad application prospects and significant advantages. This method breaks through the limitations of traditional assessment technology in light field image quality assessment, can accurately handle multi-view, complex distortion types and high-dimensional data characteristics, and is suitable for fields with high requirements on light field image quality, such as virtual reality (VR), augmented reality (AR), computational photography and medical imaging. Through cross-coding technology, the geometric relationship of multi-view information in light field images is fully utilized, avoiding the limitations of single-view assessment in traditional methods, thereby improving the accuracy and comprehensiveness of the assessment.

[0072] In addition, the present invention automatically selects the most relevant expert network according to the input features through a dynamic expert selection (MoE) mechanism. This design enables the model to flexibly adjust the evaluation strategy according to the specific distortion type and quality dimension of the light field image. Compared with the limitations of fixed weight allocation in traditional methods, the MoE mechanism can effectively optimize the utilization of computing resources and improve evaluation efficiency while ensuring high evaluation accuracy. Combined with the Pyramid Squeeze Attention (PSA) and Spatial and Channel Synergistic Attention (SCSA) modules, the present invention can enhance the model's sensitivity to light field image details and structures when processing local and global geometric distortions, effectively reduce redundant information interference, and thus enhance the robustness and generalization ability of the model.

[0073] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for blind light field image quality assessment based on dynamic expert selection, characterized in that: include: Generate N groups of cross-coded grayscale images from the light field sub-aperture images; Extracting features in the horizontal direction and the vertical direction for the N groups of cross-coded grayscale images to obtain features in the horizontal direction and features in the vertical direction; Perform local feature aggregation on the horizontal and vertical features to obtain several cross-combination features; After several cross-combined features are spliced ​​together, global feature aggregation is performed to obtain aggregated features; The aggregated features are combined with dynamic expert selection, and the two most relevant experts are dynamically selected according to probability modeling. The quality assessment score of the light field image is obtained by selecting a combination of experts to complete the quality scoring.

2. The method for blind light field image quality assessment based on dynamic expert selection as claimed in claim 1, characterized in that: The array size of the light field sub-aperture image is n×n.

3. The method for blind light field image quality assessment based on dynamic expert selection as claimed in claim 1, characterized in that: Generating N groups of cross-coded grayscale images from the light field sub-aperture images includes: Selecting target images from the light field sub-aperture image array according to different diagonal directions, wherein each diagonal direction includes a plurality of horizontal and vertical cross combinations to form a cross-coded image combination, and each of the cross-coded image combinations is composed of the same number of horizontal sub-aperture images and vertical sub-aperture images; The horizontal sub-aperture images are stacked in the horizontal direction, and the vertical sub-aperture images are stacked in the vertical direction to generate N groups of cross-coded grayscale images.

4. The method for blind light field image quality assessment based on dynamic expert selection as claimed in claim 3, characterized in that: The different diagonal directions are respectively a left diagonal and a right diagonal.

5. The method for blind light field image quality assessment based on dynamic expert selection as claimed in claim 1, characterized in that: Obtaining horizontal and vertical features includes: By utilizing the sliding window mechanism and the self-attention mechanism, multi-scale features are extracted layer by layer from the cross-coded grayscale image to capture the detail changes of the local area in the light field image and the geometric associations between different perspectives, thereby obtaining horizontal and vertical features.

6. The method for blind light field image quality assessment based on dynamic expert selection as claimed in claim 1, characterized in that: The features of obtaining several cross combinations include: Perform multi-scale decomposition on the horizontal and vertical features respectively to extract local geometric information at different spatial scales; The local geometric information of different scales is fused layer by layer to capture the local geometric distortion patterns in the multi-view information of the light field image and obtain several cross-combination features.

7. The method for blind light field image quality assessment based on dynamic expert selection as claimed in claim 1, characterized in that: The aggregate features obtained include: The spatial attention mechanism is used to process the spliced ​​cross-combined feature maps. The key areas and global structural information in the image are obtained by calculating the importance weight of each spatial position in the feature map. The channel attention mechanism is used to process the concatenated cross-combined feature maps. By calculating the importance weight of each channel, the important channel information in the feature map is highlighted, the redundant channel information is suppressed, and the aggregated features are obtained.

8. The method for blind light field image quality assessment based on dynamic expert selection as claimed in claim 1, characterized in that: The quality assessment scores for obtaining light field images include: Calculate the selection probability of each expert based on the aggregated features through a gating network, and select the two experts with the highest weights; Each expert focuses on different quality dimensions of light field images, and the outputs of the two experts are weighted fused to obtain the quality assessment score of the light field image.

Images