A full-reference light field image quality assessment method from sparse to dense viewpoints

Abstract

The present invention relates to a full-reference light field image quality evaluation method from sparse to dense viewpoints, comprising: obtaining a plurality of sub-aperture image arrays of different sizes, selecting a plurality of sub-aperture images from the sub-aperture image arrays and processing them, obtaining the Y channel information of the original and distorted viewpoint image stacks; extracting features of the Y channel information of the original and distorted viewpoint image stacks, obtaining the complete original image stack features and the complete distorted image stack features, and obtaining the contrast features; extracting image attention information from the complete original image stack features, the complete distorted image stack features, and the contrast features; extracting light field image features after rearranging the dimensions of the image attention information; obtaining final feature vector information based on the light field image features, and obtaining the final quality score based on the final feature vector information. The present invention improves the quality of full-reference light field image quality evaluation.

Description

Technical Field

[0001] The present invention relates to the technical field of quality evaluation in image processing technology, and in particular to a method for evaluating the quality of a full-reference light field image from sparse to dense viewpoints. Background Art

[0002] As a multi-dimensional computational imaging technology, light field imaging technology can simultaneously record the light intensity distribution and angle information of the imaging plane, and is considered to be the next generation of imaging technology. Light field images record the intensity and direction information of all light rays reaching a specific location from different directions. Compared with traditional two-dimensional and three-dimensional images, light field images provide richer scene information. These unique advantages make light field technology have broad application prospects in virtual reality, augmented reality, computational photography, and medical imaging. For example, light field images captured by light field cameras can realize functions such as free viewing angle, depth of field adjustment, focus reselection, and 3D modeling in post-processing. Although the multi-view and multi-dimensional information of light field images have brought new opportunities for image processing and applications, they have also brought significant technical challenges. Among them, light field image quality assessment (LFIQA) is a key issue that needs to be solved urgently. Traditional image quality assessment methods are usually based on the human visual system model to evaluate the distortion and visual quality of two-dimensional images. These methods often show limitations when processing light field images. This is mainly because light field images contain not only the basic information of two-dimensional images, but also the direction and spatial distribution information of light, which makes its quality evaluation more complicated.

[0003] At present, the methods for evaluating the quality of light field images are mainly divided into two categories: subjective evaluation and objective evaluation. Objective evaluation methods mainly include evaluation models based on feature extraction, deep learning, and comprehensive multi-view information. Methods based on feature extraction usually evaluate image quality by extracting features such as texture, edge, and structure of light field images. However, due to the high dimensionality and complexity of light field images, these feature extraction methods show certain limitations when processing multi-view and depth information. Deep learning methods can automatically learn the quality evaluation criteria of light field images by constructing neural network models and training them with large-scale light field image data. However, these methods often require a lot of computing resources and training data, and in practical applications, the generalization ability and robustness of deep learning models still need to be further verified. The evaluation model that integrates multi-view information attempts to combine the information of each view of the light field image to build a more comprehensive quality evaluation model. These methods can better reflect the overall visual effect of the light field image, but at the same time they also increase the complexity and computational complexity of the model. Summary of the invention

[0004] The purpose of the present invention is to provide a full-reference light field image quality evaluation method from sparse to dense viewpoints, deeply explore the spatial correlation of light field image subaperture array images and make full use of the visual characteristics of the human eye to improve the scoring quality.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A full-reference light field image quality assessment method from sparse to dense viewpoints, comprising:

[0007] Acquire a plurality of sub-aperture image arrays of different sizes, select a plurality of sub-aperture images from the sub-aperture image arrays and process them respectively, and acquire Y channel information of an original viewpoint image stack and Y channel information of a distorted viewpoint image stack;

[0008] Performing feature extraction on the Y channel information of the original viewpoint image stack and the Y channel information of the distorted viewpoint image stack to obtain complete original image stack features and complete distorted image stack features, and obtaining contrast features between the complete original image stack features and the complete distorted image stack features;

[0009] Extracting image attention information from the complete original image stack features, the complete distorted image stack features, and the contrast features;

[0010] After rearranging the dimensions of the image attention information, extracting light field image features;

[0011] Based on the light field image features, final feature vector information is obtained, and based on the final feature vector information, a final quality score is obtained.

[0012] Optionally, processing the sub-aperture image includes:

[0013] Selecting a plurality of sub-aperture images from the sub-aperture image array to obtain a new sub-aperture image array;

[0014] Based on the horizontal direction, a plurality of viewpoint images are selected from the new sub-aperture image array for stacking to obtain a viewpoint image stack;

[0015] The viewpoint image stack is randomly cropped, probabilistically rotated, and scaled for preprocessing, and the preprocessed viewpoint image stack is converted from the RGB space to the YUV space to obtain the Y channel information of the original viewpoint image stack and the Y channel information of the distorted viewpoint image stack.

[0016] Optionally, obtaining the complete original image stack features and the complete distorted image stack features includes:

[0017] Inputting the Y channel information of the original viewpoint image stack and the Y channel information of the distorted viewpoint image stack into a VIT feature extraction network to obtain original image stack features and distorted image stack features;

[0018] The feature dimensions of the original image stack features and the distorted image stack features are transformed by using the rearrange function, and the dimension-transformed original image stack and the distorted image stack are cascaded to obtain the complete original image stack features and the complete distorted image stack features.

[0019] Optionally, obtaining a contrast feature between the complete original image stack feature and the complete distorted image stack feature includes: inputting the complete original image stack feature and the complete distorted image stack feature into a feature contrast network to perform contrast feature amplification to obtain the contrast feature;

[0020] The method for amplifying the contrast feature is:

[0021]

[0022] Where e represents the magnified contrast feature between the complete original image stack feature and the complete distorted image stack feature, I r Represents the complete original image stack features, I d represents the complete distorted image stack features, as a parameter.

[0023] Optionally, before extracting image attention information from the complete original image stack features, the complete distorted image stack features and the contrast features, the method includes: performing feature dimensionality reduction on the complete original image stack features, the complete distorted image stack features and the contrast features using dimensionality reduction convolution.

[0024] Optionally, extracting the image attention information includes:

[0025] The complete original image stack features, the complete distorted image stack features and the contrast features are input into a coordinate-aware residual network to extract the image attention information, wherein the coordinate-aware residual network is constructed based on embedding a coordinate attention module in a residual skip connection.

[0026] Optionally, the coordinate-aware residual network extracting the image attention information includes:

[0027] Input the complete original image stack features, the complete distorted image stack features and the contrast feature crop input features after dimensionality reduction into X average pooling and Y average pooling, and perform adaptive average pooling on the height and width respectively to obtain the pooled features;

[0028] After concatenating and reducing the dimension of the pooled features, they are input into the BatchNorm layer to obtain the reduced-dimensional features;

[0029] Activate the reduced-dimensional features to obtain activated features, perform dot multiplication on the reduced-dimensional features and the activated features to obtain dot-multiplied features;

[0030] The feature after the dot product is split in a preset dimension by a split function to obtain a first feature and a second feature;

[0031] The second feature is transposed, the second feature after the transposition operation and the first feature are input into the dimensionality-raising convolution and activated by the sigmoid function, and the activated first feature, the second feature and the input feature are dot-multiplied to obtain the image attention information.

[0032] Optionally, rearranging the dimensions of the image attention information includes: rearranging the dimensions of the image attention information using a rearrange function.

[0033] Optionally, based on the light field image feature, obtaining final feature vector information includes:

[0034] Rearranging the light field image features to obtain dimensionally transformed features, wherein the light field image features are extracted based on a SwinTransformer network;

[0035] Cascading the features after the dimension conversion to obtain cascade features;

[0036] The rearrange function is used to rearrange the cascade features, and a slicing operation is performed on the rearranged cascade features to obtain the final feature vector information.

[0037] Optionally, obtaining the final quality score includes:

[0038] The final feature vector information is input into a quality regression network to obtain the final quality score, wherein the quality regression network is composed of a Linear fully connected layer and a GELU activation function.

[0039] The beneficial effects of the present invention are:

[0040] Since light field images contain not only the basic information of two-dimensional images, but also the direction and spatial distribution information of light, their quality evaluation becomes more complicated. The method proposed in the present invention can use the representation form of sub-aperture image stacks from sparse to dense light field images, which is more in line with the visual characteristics of the human eye and improves the scoring effect of full-reference light field image quality evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A flow chart of a method for evaluating the quality of a full-reference light field image from sparse to dense viewpoints according to an embodiment of the present invention;

[0043] Figure 2 A framework diagram of a method for evaluating the quality of a full-reference light field image from sparse to dense viewpoints according to an embodiment of the present invention;

[0044] Figure 3 It is a coordinate attention module in the coordinate-aware residual network of an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] This embodiment provides a full-reference light field image quality evaluation method from sparse to dense viewpoints, including:

[0048] Acquire a plurality of sub-aperture image arrays of different sizes, select a plurality of sub-aperture images from the sub-aperture image arrays and process them respectively, and acquire Y channel information of an original viewpoint image stack and Y channel information of a distorted viewpoint image stack;

[0049] Furthermore, the sub-aperture image is processed including: selecting a number of sub-aperture images from the sub-aperture image array to obtain a new sub-aperture image array; based on the horizontal direction, selecting a number of viewpoint images from the new sub-aperture image array for stacking to obtain a viewpoint image stack; randomly cropping the viewpoint image stack, probabilistically rotating it, scaling it for preprocessing, and converting the preprocessed viewpoint image stack from the RGB space to the YUV space to obtain the Y channel information of the original viewpoint image stack and the Y channel information of the distorted viewpoint image stack.

[0050] Specifically, nine representative sub-aperture images are respectively selected from the sub-aperture image arrays of four light field images of different sizes ranging from 9×9 to 3×3. The specific process includes:

[0051] The dimension of the light field subaperture array image is m×n×3×h×w. Sparse to dense viewpoint (9×9, 7×7, 5×5, 3×3) subaperture array images are extracted from the subaperture array image. Nine representative viewpoints are selected from the four viewpoint subaperture array images and stacked in groups of three to form view stacks, I1, I2, I3, and I4. Its dimension is m×n×3×h×w, where h×w are the two spatial resolutions of a single subaperture image in the subaperture array image I, m×n are the two angular resolutions of the subaperture array image I, and 3 is the number of images stacked in the image stack I.

[0052] The selected sub-aperture image is randomly cropped, and operations such as flipping, scaling, and rotating are performed with probability; the preprocessed specific sub-aperture image array of the light field image is converted from the RGB space to the YUV space, and the Y channel information of the specific sub-aperture image array is extracted. The specific process includes:

[0053] Firstly, the selected sub-aperture images are randomly cropped from the acquired sub-aperture array view stack, from 434×625 to 224×224, and flipped, scaled, and rotated probabilistically; the sub-aperture image stack I is converted from RGB space to YUV space, and then the Y channel information is extracted to obtain a view stack with a spatial resolution of 224×224.

[0054] Extracting features from the Y channel information of the original viewpoint image stack and the Y channel information of the distorted viewpoint image stack to obtain features of the complete original image stack and features of the complete distorted image stack, and obtaining contrast features between the features of the complete original image stack and the features of the complete distorted image stack;

[0055] Furthermore, obtaining the complete original image stack features and the complete distorted image stack features includes: inputting the original viewpoint image stack Y channel information and the distorted viewpoint image stack Y channel information into the VIT feature extraction network to obtain the original image stack features and the distorted image stack features; using the rearrange function to transform the feature dimensions of the original image stack and the distorted image stack, cascading the dimensionally transformed original image stack and the distorted image stack, and obtaining the complete original image stack features and the complete distorted image stack features.

[0056] Specifically, the Y channel information of the original and distorted sub-aperture image stacks is obtained, with a dimension of (m·n)×h×w. The feature learning network used is the Vision Transformer pre-trained on the ImageNet-21k dataset. The model weight is vit_base_patch16_224_in21k. The model accepts 224x224 input, PatchSize is 16x16, and the image is divided into 16×16 small blocks. Since the network needs to obtain features, the classification head MLP behind the network is removed, and the semantic feature dimension of the original and distorted viewpoint stacks is 1×196×768. The semantic information of each stack of the original image and distorted image after VIT feature extraction is cascaded to obtain the complete original image stack features and the complete distorted image stack features;

[0057] Furthermore, obtaining contrast features between the complete original image stack features and the complete distorted image stack features includes: inputting the complete original image stack features and the complete distorted image stack features into a feature contrast network to amplify the contrast features and obtain the contrast features.

[0058] Specifically, the dimensions of the obtained complete original image stack features and complete distorted image stack features are rearranged from b(h×w)c to b×c×(h×w), where h×w is the spatial resolution of the original and distorted image stack features. C is the number of channels, and B is the batch size. Then the rearranged complete original stack features and complete distorted image stack features are sent to the feature contrast network for subtraction to obtain the contrast features between the complete original image stack features and the complete distorted image stack features. The feature contrast network uses three dimensionality reduction convolutions with a 1x1 kernel size, a step size of 1, and a padding value of 0 to reduce the 2304 dimension to 768. The dimension at this time is 1×768×14×14.

[0059] Image attention information is extracted from the complete original image stack features, the complete distorted image stack features and the contrast features;

[0060] Furthermore, before extracting image attention information from the complete original image stack features, the complete distorted image stack features and the contrast features, the following steps are performed: using dimensionality reduction convolution to perform feature dimensionality reduction on the complete original image stack features, the complete distorted image stack features and the contrast features.

[0061] Furthermore, extracting image attention information includes: inputting complete original image stack features, complete distorted image stack features and contrast features into a coordinate-aware residual network to extract image attention information, wherein the coordinate-aware residual network is constructed based on embedding a coordinate attention module in a residual jump connection.

[0062] Furthermore, the coordinate-aware residual network extracts image attention information including:

[0063] The complete original image stack features, the complete distorted image stack features and the contrast feature crop input features after dimensionality reduction are input into the X average pool and the Y average pool, and the height and width are adaptively averaged and pooled respectively to obtain the pooled features; the pooled features are cascaded and dimensionally reduced, and then input into the BatchNorm layer to obtain the reduced dimensionality features; the reduced dimensionality features are activated to obtain the activated features, and the reduced dimensionality features and the activated features are dot-multiplied to obtain the dot-multiplied features; the dot-multiplied features are split in the preset dimension by the split function to obtain the first feature and the second feature; the second feature is transposed, and the second feature and the first feature after the transposition operation are input into the dimensionality-increasing convolution and activated by the sigmoid function, and the activated first feature, the second feature and the input feature are dot-multiplied to obtain the image attention information.

[0064] Specifically, the complete original image stack features, the complete distorted image stack features, and the contrast features are respectively sent to the coordinate-aware residual network. The coordinate-aware residual network is obtained by adding an attention mechanism to the residual network, and the attention used is the coordinate attention. After the coordinate-aware residual network, the complete original image stack features, the complete distorted image stack features, and the contrast feature attention information are obtained. The dimension at this time is 1×768×14×14.

[0065] The dimensions of the image attention information are rearranged and input into the Swin Transformer network to extract the light field image features;

[0066] Furthermore, rearranging the dimensions of the image attention information includes: rearranging the dimensions of the image attention information using a rearrange function.

[0067] Specifically, the dimensions of the original, distorted and contrast images are rearranged from bc(c1,c2,c3)hw, c1 is 3, c2 is 16, c3 is 16, and the rearranged dimensions are b c1(c2 h)(c3w), 1×3×224×224. The rearranged original, distorted and contrast attention features are then fed into the pre-trained Swin Transformer respectively. The Swin learning network used is the basic version of the Swin Transformer model using 4x4 patch partitioning, 7x7 window and 224x224 input size. The three final sizes are 1×49×1024, b(hw)c.

[0068] Based on the light field image features, final feature vector information is obtained, and based on the final feature vector information, a final quality score is obtained.

[0069] Furthermore, based on the light field image features, obtaining the final feature vector information includes:

[0070] Rearrange the light field image features to obtain dimensionally transformed features;

[0071] Cascade the features after dimension conversion to obtain cascade features;

[0072] The rearrange function is used to rearrange the cascade features, and the slice operation is performed on the rearranged cascade features to obtain the final feature vector information.

[0073] Specifically, the dimensions of the acquired light field image features are rearranged from b×(h×w)×C to b×c×h×w. 1×1024×7×7. Then the three feature maps are cascaded in the channel dimension to obtain a feature map of size 1×3072×7×7. The features are rearranged from the shape of 1×3072×7×7 to 1×49×3072 using the rearrange function. The arranged features are sliced ​​to extract a specific one-dimensional slice from the original four-dimensional tensor, i.e., the C channel dimension. A feature vector of dimension 1×3072 is obtained.

[0074] Furthermore, obtaining the final quality score includes:

[0075] The final feature vector information is input into the quality regression network to obtain the final quality score, where the quality regression network consists of a Linear fully connected layer and a GELU activation function.

[0076] Specifically, the 1×3072 final feature vector information is fed into the quality regression network, which consists of two fully connected linear layers and a GELU activation function. The linear layer is used to regress the quality score.

[0077] The following is based on the attached Figure 1-Figure 2 The method of the present invention is further described:

[0078] (1) Select 9 representative viewpoints from the 9×9, 7×7, 5×5, and 3×3 sub-aperture image arrays to obtain 4 sub-aperture image arrays containing 9 viewpoints. Select every 3 viewpoint images for stacking in a certain horizontal direction to obtain a viewpoint image stack. Then the viewpoint image stack is randomly cropped, rotated and scaled with probability, and finally converted from RGB space to YUV space, and the Y channel information of the pre-processed viewpoint image stack is extracted. In this embodiment, the dimensions of the processed viewpoint image stack are 9×9×9×224×224, 7×7×9×224×224, 5×5×9×224×224, and 3×3×9×224×224. Different viewpoint image stacks contain 3 groups of image stacks composed of 3 images stacked. The dimension of the Y channel information of the extracted image stack is 1×3×224×224.

[0079] (2) The Y channel information of the original and distorted viewpoint image stacks (the original 3 groups and the distorted 3 groups are image stacks composed of 3 images) are sent to the VIT feature extraction network, which includes 6 identical VIT modules, and 3 groups of original image stacks and 3 groups of distorted image stack features are obtained, and their dimensions are all 1×196×768. The 6 groups of features of the original image stacks and distorted image stacks are transformed using the rearrange function, and the dimensions of 1×196×768 (b, (h×w), c) are converted to 1×768×14×14 (b, c, h, w). Then, three sets of single sets of original image stack features with dimensions of 1×768×14×14 and three sets of single sets of distorted image stack features with dimensions of 1×768×14×14 are cascaded to obtain a complete original image stack feature with dimensions of 1×2304×14×14 and a complete distorted image stack feature with dimensions of 1×2304×14×14.

[0080] (3) The complete original image stack features and the distorted image stack features obtained in (2) are sent to the feature contrast network. The feature contrast network uses the following formula to perform formula subtraction and amplification contrast features on the complete original image stack features and the distorted image stack features.

[0081]

[0082] Where e represents the magnified contrast feature between the complete original image stack and the distorted image stack, I r Represents the complete original image stack features, I dRepresents the complete distorted image stack feature, the default setting parameter ε = 1, and the obtained contrast feature dimension is 1 × 2304 × 14 × 14. Then the complete original image stack feature, the complete distorted image stack feature, and the contrast feature will go through a layer of dimensionality reduction convolution with a convolution kernel size of 1, a step size of 1, and a padding value of 0, reducing the input feature from 2304 of dimension 1 to 768.

[0083] (4) The complete original image stack features, the complete distorted image stack features, and the contrast features obtained in step (3) are respectively fed into the coordinate-aware residual network. The proposed coordinate-aware residual network embeds a coordinate attention module in the traditional residual skip connection, such as Figure 3 As shown. The coordinate attention module used first sends the input features to "X average pooling" and "Y average pooling" respectively. "X average pooling" and "Y average pooling" are both average pooling layers, which perform adaptive average pooling on the height and width respectively to obtain the pooled features x_h and x_w, with dimensions of 1×768×14×1. Then x_h and x_w are cascaded in the second dimension to obtain feature y, with dimensions of 1×768×28×1. Then feature y is sent to a layer of dimensionality reduction convolution layer with a convolution kernel size of 1, a step size of 1, and a padding value of 0, which reduces the first dimension of feature y from 768 to 24, and then passes through a layer of BatchNorm layer to obtain the reduced feature y, with dimensions of 1×24×28×1. The reduced feature y is sent to the h_sigmoid layer, which uses the ReLU6 activation function to implement h-sigmoid (ReLU6 is a ReLU activation function that limits the input value to between 0 and 6). h-sigmoid activation function: This function is implemented by adding a constant 3 to the input value, then passing it through a ReLU6 activation function, and finally dividing it by 6. The purpose is to map the input value range from [-∞, +∞] to [0, 1], simulating the S-shaped curve of the sigmoid function to obtain the feature y after activation by the h_sigmoid layer, and then multiplying the activated feature y with the input feature y after dimensionality reduction to obtain the feature y1 after the dot product, with a dimension of 1×24×28×1.

[0084] The feature y1 is split into two parts in the second dimension (i.e., the height dimension) by the split function, and the heights are h=14 and w=14 respectively. x_h will contain the part with height h, i.e., the first feature, and x_w will contain the part with height w, i.e., the second feature. At this time, both x_h and x_w are 1×24×14×1. Then, x_w is transposed to change its dimension from 1×24×14×1 to 1×24×1×14. x_h and x_w are sent to a layer of convolution with a kernel size of 1, a step size of 1, and a padding value of 0 to increase the second dimension of x_h and x_w from 24 to 768. Then, the sigmoid function is used to activate x_h and x_w, and the dimensions of a_h are 1×768×14×1 and a_w are 1×768×1×14. Finally, the input feature, a_h and a_w are dot-multiplied to obtain the final input attention feature with a dimension of 1×768×14×14.

[0085] (5) The original image stack features, the complete distorted image stack features, and the attention information of the contrast features obtained in step (4) are rearranged using the rearrange function. bc(c1 c2 c3)hw represents the shape of the original feature (1×768×14×14), where b is the batch size, c1=3, c2=16, and c3=16 are the three parts of the number of channels, and h and w are the height and width, respectively. b c1(c2 h)(c3 w) represents the shape of the target feature, where c1 is directly extracted, c2 is combined with h, and c3 is combined with w, resulting in a feature dimension of 1×3×224×224. The re-arranged original image stack features, the complete distorted image stack features, and the contrast features are respectively fed into the Swin feature extraction network. The Swin feature extraction network used is a basic version of the Swin Transformer model using 4x4 patch partitioning, 7x7 window and 224x224 input size, and the feature dimensions are all 1×49×1024.

[0086] (6) The original image stack features, the complete distorted image stack features, and the contrast features extracted by the Swin feature extraction network in step (5) are rearranged using the rearrange function to convert the original feature dimension of 1×49×1024 (b, (h×w), c) to 1×1024×7×7 (b, c, h, w). Then, the three features after the dimension conversion are cascaded in the second dimension to obtain a feature dimension of 1×3072×7×7. The features are rearranged from a shape of 1×3072×7×7 to 1×49×3072 using the rearrange function, and the rearranged features are sliced ​​to select the value of the first channel in each batch to obtain a feature vector with a dimension of 1×3072. The obtained feature vector is sent to the quality regression network, which consists of two Linear fully connected layers and a GELU activation function. First, the feature vector passes through the first fully connected layer, and its dimension changes from 3072 to 3072, then passes through the GeLU activation function, and finally passes through the second fully connected layer, and its dimension changes from 3072 to 1 to obtain the final quality score.

[0087] The full-reference light field image quality evaluation method proposed in the present invention makes full use of the spatial correlation between sub-aperture array images of light field images of different sizes and the visual characteristics of the human eye. The effect of full-reference light field image quality evaluation can be improved. The method proposed in the present invention first selects 9 representative sub-aperture images from the sub-aperture image arrays of 4 different sizes of light field images from 9×9 to 3×3; then the selected sub-aperture images are randomly cropped, and operations such as flipping, scaling and rotating are performed with probability; the pre-processed specific sub-aperture image array of the light field image is converted from RGB space to YUV space, and the Y channel information of the specific sub-aperture image array is extracted. Subsequently, the obtained Y channel information of the original and distorted view stack images is sent to the VIT feature extraction network to obtain a single set of image stack features. Then, the feature information of the original and distorted view stack images is cascaded respectively, and the cascaded complete original image stack features and the complete distorted image stack features are subtracted to obtain contrast features. Then, the complete original image stack features, the complete distorted image stack features and the contrast features are sent to the coordinate-aware residual network. Finally, the extracted features are sent to the quality regression network to obtain the final quality score. Compared with the traditional light field image quality evaluation algorithm, the present invention can deeply explore the spatial correlation of the sub-aperture array image of the light field image and make full use of the visual characteristics of the human eye, thereby improving the scoring quality.

[0088] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A full-reference light field image quality assessment method from sparse to dense viewpoints, characterized in that: include: Acquire a plurality of sub-aperture image arrays of different sizes, select a plurality of sub-aperture images from the sub-aperture image arrays and process them respectively, and acquire Y channel information of an original viewpoint image stack and Y channel information of a distorted viewpoint image stack; Extracting features of the Y channel information of the original viewpoint image stack and the Y channel information of the distorted viewpoint image stack to obtain features of a complete original image stack and features of a complete distorted image stack, and obtaining contrast features between the features of the complete original image stack and the features of the complete distorted image stack; Extracting image attention information from the complete original image stack features, the complete distorted image stack features, and the contrast features; After rearranging the dimensions of the image attention information, extracting light field image features; Based on the light field image features, final feature vector information is obtained, and based on the final feature vector information, a final quality score is obtained.

2. The method for evaluating image quality of a full-reference light field from sparse to dense viewpoints according to claim 1, characterized in that: Processing the sub-aperture image includes: Selecting a plurality of sub-aperture images from the sub-aperture image array to obtain a new sub-aperture image array; Based on the horizontal direction, a plurality of viewpoint images are selected from the new sub-aperture image array for stacking to obtain a viewpoint image stack; The viewpoint image stack is randomly cropped, probabilistically rotated, and scaled for preprocessing, and the preprocessed viewpoint image stack is converted from the RGB space to the YUV space to obtain the Y channel information of the original viewpoint image stack and the Y channel information of the distorted viewpoint image stack.

3. The full-reference light field image quality assessment method from sparse to dense viewpoints according to claim 1, characterized in that: Obtaining the complete original image stack features and the complete distorted image stack features includes: Inputting the Y channel information of the original viewpoint image stack and the Y channel information of the distorted viewpoint image stack into a VIT feature extraction network to obtain original image stack features and distorted image stack features; The feature dimensions of the original image stack features and the distorted image stack features are transformed by using the rearrange function, and the dimension-transformed original image stack and the distorted image stack are cascaded to obtain the complete original image stack features and the complete distorted image stack features.

4. The method for evaluating image quality of a full-reference light field from sparse to dense viewpoints according to claim 1, characterized in that: Acquiring contrast features between the complete original image stack features and the complete distorted image stack features comprises: inputting the complete original image stack features and the complete distorted image stack features into a feature contrast network to perform contrast feature amplification to acquire the contrast features; The method for amplifying the contrast feature is: Where e represents the magnified contrast feature between the complete original image stack feature and the complete distorted image stack feature, I r Represents the complete original image stack features, I d represents the complete distorted image stack feature, and ε is a parameter.

5. The method for evaluating image quality of a full-reference light field from sparse to dense viewpoints according to claim 1, characterized in that: Before extracting image attention information from the complete original image stack features, the complete distorted image stack features and the contrast features, the method includes: performing feature dimensionality reduction on the complete original image stack features, the complete distorted image stack features and the contrast features using dimensionality reduction convolution.

6. The method for evaluating image quality of a full-reference light field from sparse to dense viewpoints according to claim 5, characterized in that: Extracting the image attention information includes: The complete original image stack features, the complete distorted image stack features and the contrast features are input into a coordinate-aware residual network to extract the image attention information, wherein the coordinate-aware residual network is constructed based on embedding a coordinate attention module in a residual skip connection.

7. The method for evaluating image quality of a full-reference light field from sparse to dense viewpoints according to claim 6, characterized in that: The coordinate-aware residual network extracts the image attention information including: Input the complete original image stack features, the complete distorted image stack features and the contrast feature crop input features after dimensionality reduction into X average pooling and Y average pooling, and perform adaptive average pooling on the height and width respectively to obtain the pooled features; After concatenating and reducing the dimension of the pooled features, they are input into the BatchNorm layer to obtain the reduced-dimensional features; Activate the reduced-dimensional features to obtain activated features, perform dot multiplication on the reduced-dimensional features and the activated features to obtain dot-multiplied features; The feature after the dot product is split in a preset dimension by a split function to obtain a first feature and a second feature; The second feature is transposed, the second feature after the transposition operation and the first feature are input into the dimensionality-raising convolution and activated by the sigmoid function, and the activated first feature, the second feature and the input feature are dot-multiplied to obtain the image attention information.

8. The method for evaluating image quality of a full-reference light field from sparse to dense viewpoints according to claim 1, characterized in that: Rearranging the dimensions of the image attention information includes: rearranging the dimensions of the image attention information using a rearrange function.

9. The method for evaluating image quality of a full-reference light field from sparse to dense viewpoints according to claim 1, characterized in that: Based on the light field image features, obtaining final feature vector information includes: Rearranging the light field image features to obtain dimensionally transformed features, wherein the light field image features are extracted based on a SwinTransformer network; Cascading the features after the dimension conversion to obtain cascade features; The rearrange function is used to rearrange the cascade features, and a slicing operation is performed on the rearranged cascade features to obtain the final feature vector information.

10. The method for evaluating image quality of a full-reference light field from sparse to dense viewpoints according to claim 1, characterized in that: Obtaining the final quality score includes: The final feature vector information is input into a quality regression network to obtain the final quality score, wherein the quality regression network is composed of a Linear fully connected layer and a GELU activation function.

Images