Non-reference light field image quality evaluation method based on sub-aperture sequence modeling

Abstract

The invention discloses a non-reference light field image quality evaluation method based on sub-aperture sequence modeling. The method comprises the following steps: S1, preprocessing a light field image to generate a light field sub-aperture sequence; s2, extracting features of each image in the light field sub-aperture sequence by adopting a Vision Transform architecture to obtain a sub-aperture feature sequence; s3, performing feature fusion on the sub-aperture feature sequence on a row scanning dimension and a column scanning dimension through a convolution Mama module to obtain global fusion features; and S4, performing mean value processing on the obtained global fusion features, and sending the processed global fusion features to a score prediction module to obtain an objective quality evaluation score. According to the method, semantic modeling in the sub-aperture images and sequence angle consistency modeling between the sub-aperture images are considered, so that the semantic features of the sub-aperture images and the sequence angle consistency features of the sub-aperture images are better captured.

Description

Technical Field

[0001] This invention belongs to the field of light field image quality assessment technology, specifically relating to a referenceless light field image quality assessment method based on sub-aperture sequence modeling. Background Technology

[0002] Light field images (LFIs) can simultaneously record the spatial position and angular direction of light rays when acquiring scene information, thus supporting advanced functions such as multi-sub-aperture image angle refocusing and sub-aperture sequence viewpoint switching. They have been widely used in computational photography, virtual reality, and 3D reconstruction. Unlike traditional 2D image quality evaluation, the quality of light field images depends not only on the spatial sharpness and texture fidelity of individual sub-aperture images but also on cross-angle characteristics such as angular consistency and parallax smoothness between sub-aperture images. In practical applications, light field images may introduce various distortions during acquisition, interpolation, compression, and reconstruction processes, such as interpolation artifacts, parallax jumps, perspective misalignment, and local flicker. These distortions significantly reduce the coherence of the sub-aperture image sequence and the overall perceived quality during light field display.

[0003] Existing light field image quality assessment methods typically focus on the extraction and fusion of two-dimensional distortion features in the spatial domain, angular domain, and outer polar geometric domain. These methods are insufficient to effectively characterize the discontinuities and perceptual inconsistencies caused by the smooth switching of sub-aperture image sequences during light field display or interactive sub-aperture image sequence browsing, such as parallax jumps and flickering between sub-aperture image angles. To improve the perceptual ability of the row and column scanning sequence consistency of sub-aperture image sequences, the model design needs to consider the following three aspects: (1) mapping the light field sub-aperture images into a coherent sequence along the row and column scanning sequence to more realistically simulate the human eye observation process; (2) introducing vector representations that can characterize global semantic information at the sub-aperture image level; and (3) adopting a sequence modeling mechanism that can simultaneously model the local smooth evolution and long-range dependencies of the sub-aperture image sequence in the row and column scanning dimension. Summary of the Invention

[0004] This invention aims to overcome the shortcomings of existing traditional methods and proposes a referenceless light field image quality evaluation method based on sub-aperture sequence modeling.

[0005] This invention employs two key technical approaches as its foundation: First, it utilizes Vision Transformer (ViT) to compress the spatial features of each sub-aperture image in the light field image into a unified vector form, generating global features of the light field sub-aperture sequence. Second, it employs the selective state-space model Mamba-2 to perform efficient state recursion and long-range modeling among the global features of the sub-aperture sequence. These two key technical approaches balance semantic modeling within sub-aperture images with sequence angular consistency modeling between sub-aperture images, thereby better capturing the semantics of sub-aperture images and the angular consistency features of the sub-aperture image sequence.

[0006] The technical solution adopted by this invention to solve its technical problem is as follows:

[0007] Step S1: Preprocess the light field image to generate a light field sub-aperture sequence;

[0008] Step S2: For each image in the light field sub-aperture sequence, extract features using the Vision Transformer (ViT) architecture to obtain the sub-aperture feature sequence;

[0009] Step S3: The sub-aperture feature sequences are fused in both row and column scan dimensions using a convolutional Mamba module to obtain global fused features.

[0010] Step S4: The obtained global fusion features are averaged and then fed into the score prediction module to obtain the objective quality evaluation score;

[0011] The specific implementation of step S1 includes:

[0012] Step S1-1: Using the dual-plane representation of the light field, each original light field distortion image in the light field image quality assessment dataset is defined as... ,in These are the angular coordinates of the original light field distortion image. Represents the spatial coordinates of the original light field distortion image; The range of values ​​is , The range of values ​​is , The range of values ​​is , The range of values ​​is That is, the resolution of the original light field distortion image is , contains a set A sub-aperture image array, where each sub-aperture image has an image resolution of [missing information]. ;

[0013] Step S1-2: Randomly crop and scale the original light field distortion image to obtain... The light field distortion image is then rotated and flipped horizontally with random probabilities of 20% and 80%, with rotation angles including 90 degrees, 180 degrees, and 270 degrees, to obtain the data-enhanced light field distortion image.

[0014] Step S1-3: Extract the 5×5 sub-aperture image array from the center of the data-enhanced light field distortion image. And scan in a "W" shape (odd columns first, then even columns, odd columns from top to bottom, even columns from bottom to top) Perform a continuous column scan, and then... The sub-aperture images in the image are arranged into a light field sub-aperture sequence with a length of 25. ;

[0015] Step S2 specifically includes:

[0016] Step S2-1: Sequence of optical field sub-apertures Each sub-aperture image is fed into a pre-trained VisionTransformer (ViT) model for feature extraction. The pre-trained ViT model divides each sub-aperture image into 16×16 image blocks, which are linearly mapped to block embedding vectors, resulting in a 196×768 block embedding vector sequence. And a 1×768 class embedding vector CLS;

[0017] Step S2-2: Pass the category embedding vector CLS of each sub-aperture image through a linear mapping layer to obtain a 1×512 compressed category embedding vector; then process the light field sub-aperture sequence... The compressed class embedding vectors of all sub-aperture images are concatenated to form a 25×512 sub-aperture feature sequence. .

[0018] The specific implementation of step S3 includes:

[0019] Step S3-1: Sub-aperture feature sequence It is sequentially divided into 5 feature subsequences of equal length. k=1, 2, 3, 4, 5; each feature column subsequence The size is 5×512.

[0020] Step S3-2: Subsequence of each feature column The data is fed into a convolutional Mamba module to perform selective state-space modeling along the column directions of the sub-aperture sequence, resulting in a feature column fusion sub-sequence. .

[0021] The convolutional Mamba module consists of a Mamba-2 module and a convolutional module, and its implementation is as follows:

[0022] Input feature sequence After undergoing layer normalization (LayerNorm) operation, the data is fed into the Mamba-2 module to establish... Long-range dependencies Then through a learnable scaling factor The jump connection will and The fusion yields a sequence that is identical to the input feature sequence. Initial feature fusion sequences of the same size Furthermore, After undergoing layer normalization (LayerNorm) operation, the data is fed into a convolutional module containing two 3×3 convolutional layers to capture... Local details Then through a learnable scaling factor The jump connection will and Fusion, output feature fusion sequence Finally, the input feature sequence will be... and By merging, the output of the convolutional Mamba module is obtained. .

[0023] The Mamba-2 module mentioned is the MambaBlock proposed in "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality".

[0024] Step S3-3: Fuse the 5 feature columns into a subsequence Concatenate them sequentially to form a 25×512 feature column sequence. ;Will The input is fed into the aforementioned convolutional Mamba module to achieve selective state-space modeling of continuous column scanning of the sub-aperture sequence, resulting in a feature column fusion sequence. .

[0025] Step S3-4: Merge the feature columns in the order of the consecutive column scans in Step S1-3. The features are rearranged into a 5×5 feature array and transposed; then, using a "W-shaped" scanning method, the transposed feature array is combined into a 25×512 feature row sequence. , The sequence order is consistent with the continuous row scanning order of the sub-aperture sequence. Replace step S3-1 Repeat steps S3-1 to S3-3 to perform selective state-space modeling for each row and consecutive rows of the sub-aperture sequence, respectively, and finally obtain the global fusion features. .

[0026] Step S4 specifically includes:

[0027] Global fusion features The mean was used for processing. The obtained mean characteristics... As input to the score prediction module, the score prediction module contains two linear layers. The first linear layer maps 1×512 features to 1×128, followed by an activation layer with GELU as the activation function. The second linear layer maps 1×128 features to 1×1, which is used as the final predicted score. The predicted score is compared with the MOS to calculate the loss function.

[0028] The beneficial effects of this invention are as follows:

[0029] 1. This invention rearranges the central 5×5 sub-aperture image array into a continuous light field sub-aperture sequence in a "W-shaped" row and column scanning order. Combined with a pre-trained Vision Transformer (ViT), features are extracted and modeled for each sub-aperture image in the sequence. This not only preserves the spatial semantic information of a single sub-aperture image, but also realizes the continuous representation of the sub-aperture sequence in the row and column scanning dimension. It can accurately simulate the continuous observation process under row and column scanning, thereby greatly improving the sensitivity and detection capability of sequence transition distortions (such as parallax jumps, sequence flicker, etc.).

[0030] 2. This invention relies on the category embedding vector CLS extracted by ViT, and obtains a compressed category embedding vector of uniform dimension through linear mapping. This vector is then used as the input of the convolutional Mamba module. Based on the selective state space modeling mechanism of the Mamba-2 module, efficient state recursion and long-range dependency modeling between sub-aperture sequence features are achieved. This fully explores the correlation features between sub-aperture images under the row and column scanning dimensions, significantly enhancing the pertinence and effectiveness of sequence modeling. Finally, it outputs more semantic and robust features for quality regression tasks. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the main steps of the present invention.

[0032] Figure 2 This is a schematic diagram of the optical field sub-aperture sequence input to the model of this invention.

[0033] Figure 3 This is a schematic diagram of the referenceless light field image quality assessment method model of the present invention, which combines Vision Transformer (ViT) and convolutional Mamba module for modeling.

[0034] Figure 4 This is a schematic diagram of the Vision Transformer network coding module of the present invention.

[0035] Figure 5 This is a structural diagram of the convolutional Mamba module of the present invention. Figure 6 For different optical field sub-aperture sequences. Detailed Implementation

[0036] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0037] like Figure 1 As shown, the referenceless light field image quality assessment method based on sub-aperture sequence modeling adopts the following technical steps:

[0038] Step S1: Preprocess the light field image to generate a light field sub-aperture sequence;

[0039] Step S2: For each image in the light field sub-aperture sequence, extract features using the Vision Transformer (ViT) architecture to obtain the sub-aperture feature sequence;

[0040] Step S3: The sub-aperture feature sequences are fused in both row and column scan dimensions using a convolutional Mamba module to obtain global fused features.

[0041] Step S4: The obtained global fusion features are averaged and then fed into the score prediction module to obtain the objective quality evaluation score;

[0042] The specific implementation of step S1 includes:

[0043] Step S1-1: Using the dual-plane representation of the light field, each original light field distortion image in the light field image quality assessment dataset is defined as... ,in These are the angular coordinates of the original light field distortion image. Represents the spatial coordinates of the original light field distortion image; The range of values ​​is , The range of values ​​is , The range of values ​​is , The range of values ​​is That is, the resolution of the original light field distortion image is , contains a set A sub-aperture image array, where each sub-aperture image has an image resolution of [missing information]. ;

[0044] Step S1-2: Randomly crop and scale the original light field distortion image to obtain... The light field distortion image is then rotated and flipped horizontally with random probabilities of 20% and 80%, with rotation angles including 90 degrees, 180 degrees, and 270 degrees, to obtain the data-enhanced light field distortion image.

[0045] Steps S1-3: As Figure 2 As shown, a 5×5 sub-aperture image array is extracted from the center of the data-enhanced light field distortion image. And scan in a "W" shape (odd columns first, then even columns, odd columns from top to bottom, even columns from bottom to top) Perform a continuous column scan, and then... The sub-aperture images in the image are arranged into a light field sub-aperture sequence with a length of 25. ;

[0046] Furthermore, step S2 specifically includes: step S2-1: as follows Figure 4 As shown, the optical field sub-aperture sequence Each sub-aperture image is fed into a pre-trained Vision Transformer (ViT) model for feature extraction. The pre-trained ViT model divides each sub-aperture image into 16×16 image blocks, which are linearly mapped to block embedding vectors, resulting in a 196×768 block embedding vector sequence. And a 1×768 class embedding vector CLS;

[0047] Step S2-2: Pass the category embedding vector CLS of each sub-aperture image through a linear mapping layer to obtain a 1×512 compressed category embedding vector; then process the light field sub-aperture sequence... The compressed class embedding vectors of all sub-aperture images are concatenated to form a 25×512 sub-aperture feature sequence. .

[0048] Furthermore, step S3 is specifically implemented by including:

[0049] Step S3-1: As Figure 3 As shown, the sub-aperture feature sequence It is sequentially divided into 5 feature subsequences of equal length. k=1, 2, 3, 4, 5; each feature column subsequence The size is 5×512.

[0050] Step S3-2: Subsequence of each feature column The data is fed into a convolutional Mamba module to perform selective state-space modeling along the column directions of the sub-aperture sequence, resulting in a feature column fusion sub-sequence. .

[0051] like Figure 4 and 5 As shown, the convolutional Mamba module consists of a Mamba-2 module and a convolutional module, and its specific implementation is as follows:

[0052] Input feature sequence After undergoing layer normalization (LayerNorm) operation, the data is fed into the Mamba-2 module to establish... Long-range dependencies Then through a learnable scaling factor The jump connection will and The fusion yields a sequence that is identical to the input feature sequence. Initial feature fusion sequences of the same size Furthermore, After undergoing layer normalization (LayerNorm) operation, the data is fed into a convolutional module containing two 3×3 convolutional layers to capture... Local details Then through a learnable scaling factor The jump connection will and Fusion, output feature fusion sequence Finally, the input feature sequence will be... and By merging, the output of the convolutional Mamba module is obtained. .

[0053] The Mamba-2 module mentioned is the MambaBlock proposed in "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality". The specific formula is:

[0054] ;

[0055] ;

[0056] ;

[0057] ;

[0058] ;

[0059] Step S3-3: Fuse the 5 feature columns into a subsequence Concatenate them sequentially to form a 25×512 feature column sequence. ;Will The input is fed into the aforementioned convolutional Mamba module to achieve selective state-space modeling of continuous column scanning of the sub-aperture sequence, resulting in a feature column fusion sequence. .

[0060] Step S3-4: Merge the feature columns in the order of the consecutive column scans in Step S1-3. The features are rearranged into a 5×5 feature array and transposed; then, using a "W-shaped" scanning method, the transposed feature array is combined into a 25×512 feature row sequence. , The sequence order is consistent with the continuous row scanning order of the sub-aperture sequence. Replace step S3-1 Repeat steps S3-1 to S3-3 to perform selective state-space modeling for each row and consecutive rows of the sub-aperture sequence, respectively, and finally obtain the global fusion features. .

[0061] Step S4 specifically includes:

[0062] Global fusion features The mean value is used. The specific formula is:

[0063] ;

[0064] The obtained mean characteristics The input to the score prediction module is used as input. The score prediction module consists of two linear layers. The first linear layer maps 1×512 features to 1×128, followed by an activation layer using GELU as the activation function. The second linear layer maps 1×128 features to 1×1, which serves as the final predicted score. The predicted score is compared with the MOS (Mean Orientation of Memory) to calculate the loss function. In practical application, the fast-converging Adam optimizer is used, the mean squared error loss function is adopted, and the initial learning rate is set to 5e-6.

[0065] Table 1 compares the method of this invention with other full-reference / no-reference optical field quality assessment methods on the Win5-LID dataset. The indicators are Spearman Rank Correlation Coefficient (SROCC) and Pearson Linear Correlation Coefficient (PLCC). Higher SROCC and PLCC indicate better method performance. The method of this invention has an SROCC that is 0.0232 higher than the second method and a PLCC that is 0.0146 higher than the second method, demonstrating the superiority of this invention.

[0066] Table 1

[0067]

[0068] To verify the contribution of the optical field sub-aperture sequence strategy, we used other optical field sub-aperture sequence processing strategies to train our network. Figure 6 Table 2 presents a comparison of different light field sub-aperture sequences using SROCC with five-fold cross-validation on the Win5-LID dataset. This demonstrates the impact of different light field sub-aperture sequences on the light field image quality assessment task.

[0069] Table 2

[0070]

[0071] To verify the contribution of the convolutional Mamba module to the dimensionality information of consecutive row and column scans of sub-aperture image features, we constructed three ablation models: a standalone ViT model, a ViT model combined with consecutive column scan processing of the convolutional Mamba module, and a ViT model combined with consecutive row scan processing of the convolutional Mamba module. Table 3 lists their performance results on the Win5-LID dataset.

[0072] Table 3

[0073]

[0074] Finally, it should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the claims.

Claims

1. A method for evaluating the quality of referenceless light field images based on sub-aperture sequence modeling, characterized in that, Includes the following steps: Step S1: Preprocess the light field image to generate a light field sub-aperture sequence; Step S2: For each image in the light field sub-aperture sequence, extract features using the Vision Transformer architecture to obtain the sub-aperture feature sequence; Step S3: The sub-aperture feature sequences are fused in both row and column scan dimensions using a convolutional Mamba module to obtain global fused features; Step S4: The obtained global fusion features are averaged and sent to the score prediction module to obtain the objective quality evaluation score.

2. The method for evaluating the quality of a referenceless light field image based on sub-aperture sequence modeling according to claim 1, characterized in that, The specific implementation of step S1 includes: Step S1-1: Using the dual-plane representation of the light field, each original light field distortion image in the light field image quality assessment dataset is defined as... ,in These are the angular coordinates of the original light field distortion image. Represents the spatial coordinates of the original light field distortion image; Step S1-2: Randomly crop and scale the original light field distortion image, and then rotate and flip the light field distortion image horizontally and vertically with random probabilities of 20% and 80%, with rotation angles including 90 degrees, 180 degrees and 270 degrees. Step S1-3: Extract the 5×5 sub-aperture image array from the center of the data-enhanced light field distortion image. And scanned in a "W" shape. The sub-aperture image array is then scanned sequentially in a continuous column scan order. The sub-aperture images in the image are arranged into a light field sub-aperture sequence with a length of 25. .

3. The method for evaluating the quality of a referenceless light field image based on sub-aperture sequence modeling according to claim 2, characterized in that, The "W-shaped" scanning method involves scanning the odd-numbered columns first, followed by the even-numbered columns, with the odd-numbered columns going from top to bottom and the even-numbered columns going from bottom to top.

4. The method for evaluating the quality of a referenceless light field image based on sub-aperture sequence modeling according to claim 2, characterized in that, Step S2 specifically includes: Step S2-1: Sequence of optical field sub-apertures Each sub-aperture image is fed into a pre-trained VisionTransformer model for feature extraction. The pre-trained ViT model divides each sub-aperture image into image patches, which are then linearly mapped to block embedding vectors, resulting in a block embedding vector sequence. and category embedding vectors (CLS); Step S2-2: Pass the category embedding vector CLS of each sub-aperture image through a linear mapping layer to obtain a compressed category embedding vector; convert the light field sub-aperture sequence... The compressed class embedding vectors of all sub-aperture images are concatenated to form the sub-aperture feature sequence. .

5. The method for evaluating the quality of a referenceless light field image based on sub-aperture sequence modeling according to claim 4, characterized in that, The specific implementation of step S3 includes: Step S3-1: Sub-aperture feature sequence It is sequentially divided into 5 feature subsequences of equal length. k = 1, 2, 3, 4, 5; Step S3-2: Subsequence of each feature column The data is fed into a convolutional Mamba module to perform selective state-space modeling along the column directions of the sub-aperture sequence, resulting in a feature column fusion sub-sequence. ; Step S3-3: Fuse the 5 feature columns into a subsequence They are concatenated sequentially into a feature column sequence. ;Will The input is fed into a convolutional Mamba module to obtain a feature column fusion sequence. ; Step S3-4: Merge the feature columns in the order of the consecutive column scans in Step S1-3. The features are rearranged into a feature array and transposed; then, using a "W-shaped" scanning method, the transposed feature array is combined into a feature row sequence. ,Will Replace step S3-1 Repeat steps S3-1 to S3-3 to perform selective state-space modeling for each row and consecutive rows of the sub-aperture sequence, respectively, and finally obtain the global fusion features. .

6. The method for evaluating the quality of a referenceless light field image based on sub-aperture sequence modeling according to claim 5, characterized in that, The convolutional Mamba module consists of a Mamba-2 module and a convolutional module, and its implementation is as follows: Input feature sequence After layer normalization, the data is fed into the Mamba-2 module to establish... Long-range dependencies Then through a learnable scaling factor The jump connection will and The fusion yields a sequence that is identical to the input feature sequence. Initial feature fusion sequences of the same size ;Will After layer normalization, the data is fed into a convolutional module containing two 3×3 convolutional layers to capture... Local details Then through a learnable scaling factor The jump connection will and Fusion, output feature fusion sequence Finally, the input feature sequence will be... and By merging, the output of the convolutional Mamba module is obtained. .

7. The method for evaluating the quality of a referenceless light field image based on sub-aperture sequence modeling according to claim 4, characterized in that, Feature row sequence The sequence order is consistent with the continuous row scanning order of the sub-aperture sequence.

8. The method for evaluating the quality of a referenceless light field image based on sub-aperture sequence modeling according to claim 6, characterized in that, Step S4 specifically includes: Global fusion features The mean characteristics are obtained by using mean processing. As input to the score prediction module; the score prediction module contains two linear layers, the first linear layer followed by an activation layer with the GELU function as the activation function, and the second linear layer as the final predicted score; the predicted score is compared with the MOS to calculate the loss function.

Images