No-reference light field image quality evaluation method based on visual converter

Abstract

The invention discloses a no-reference light field image quality evaluation method based on a visual converter, and the method comprises the steps: extracting a sub-aperture image array of a distorted light field image, and extracting a central sub-aperture image and a sub-aperture image array of which the central angle resolution is a * a; inputting a central sub-aperture image of the distorted light field image into a space blocking module to obtain a distorted light field image space block rich in space information, and inputting a sub-aperture image array of which the central angle resolution is a * a of the distorted light field image into an angle blocking module to obtain a distorted light field image angle block; obtaining multi-layer features on a spatial domain of the distorted light field image, obtaining multi-layer features on an angle domain of the distorted light field image, and splicing the multi-layer features to obtain fusion features rich in spatial features and angle features; and carrying out regression on the fusion features by utilizing a quality score prediction network to obtain a quality score of the distorted light field image, and training the network by utilizing gradient back propagation according to a configured related network model and training parameters.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and computer vision quality evaluation, and in particular to a reference-free light field image quality evaluation method based on a vision transformer. Background Art

[0002] Nowadays, with the rapid development of immersive imaging technology, light field, stereo images and stereo videos can bring an immersive feeling that traditional two-dimensional images cannot provide. With the emergence of handheld light field cameras, light field imaging has attracted widespread attention from academia and industry. Light field cameras insert a microlens array between the main lens and the image sensor to capture the intensity and direction of light in the scene being photographed, generating light field images with rich four-dimensional spatial angle visual information. In the process from the acquisition end to the visualization end, various quality perception distortions will inevitably be introduced into the light field image, which will reduce the visual quality of the light field image. A good light field image quality evaluation method can bring people a more pleasant light field visual experience. The subjective evaluation method of light field image quality is to directly collect the evaluation scores of human visual perception quality of light field images. The subjective evaluation method is widely regarded as the most reliable and accurate method for evaluating the perceived quality of light field images. However, collecting subjective evaluation scores is a task that requires a lot of manpower and material resources. In addition, in the actual light field image evaluation application scenario, it is difficult to obtain the original light field image before being interfered by noise and distortion. Therefore, it is very necessary to design an objective reference-free light field image quality evaluation method.

[0003] Reference-free light field image quality assessment methods can be mainly divided into traditional methods and deep learning-based methods. Traditional reference-free light field image quality assessment methods are mainly based on various feature extraction and measurement methods commonly used in traditional two-dimensional image and stereoscopic image quality assessment methods. Traditional reference-free light field image quality assessment methods usually first perform image enhancement and other preprocessing on the input light field image, then convert the color space of the preprocessed light field image to reduce its number of channels, and then extract useful perceptual features from the processed light field image, and perform quality score regression on the extracted perceptual features through a multi-layer perceptron. Traditional reference-free light field image quality assessment methods have difficulty in effectively characterizing the distortion and inconsistency of light field images, and cannot accurately perceive the quality of the spatial and angular information contained in the light field image.

[0004] With the rapid development of deep learning, convolutional neural networks have become the dominant architecture in reference-free light field image quality assessment methods. These methods use convolutional neural networks to learn features directly from light field images and use fully connected layers to regress to evaluation scores. Although parallel neural networks can extract spatial and angular features of light field images separately, they not only ignore the fact that the spatial resolution is much higher than the angular resolution on the acquisition end, but also ignore the importance of angular features in stereoscopic perception. At the same time, the high performance and generalization ability of convolutional neural networks depend on the quality and quantity of light field image quality assessment datasets obtained through subjective evaluation. However, the existing light field image quality assessment datasets are smaller than the two-dimensional image quality assessment datasets, and light field image methods based on convolutional neural networks find it difficult to perceive the correct geometric features from light field images.

[0005] Prior art A reference-free light field image quality evaluation method and system based on deep learning (CN115147836A) predicts image quality by preprocessing light field images and using convolutional neural networks to learn angle features, spatial features, and angle-space fusion features. However, it ignores the gap between angle resolution and spatial resolution in light field images, and only extracts features from a preprocessed pixel block, making it difficult to correctly extract angle features. Limited by the problem of a small light field image quality evaluation data set, its performance and generalization ability are poor. The prior art only extracts spatial features and angle features simultaneously through a two-stream convolutional neural network, which not only does not fully consider the gap between spatial resolution and angle resolution at the acquisition end, but also ignores the importance of angle features in stereoscopic perception. In addition, in order to reduce the complexity of light field images, the prior art will lose part of the color information of the light field images, introduce noise and distortion when reducing complexity, and make it difficult for the evaluation method to accurately fit the subjective evaluation score. Therefore, designing a model that can perform well on a small light field image quality evaluation dataset, pay more attention to extracting angular features, effectively learn the geometric distortion representation of light field images, and fully retain color information during the preprocessing process is a key challenge that needs to be urgently addressed. Summary of the invention

[0006] In order to overcome and solve the aforementioned technical problems, the present invention provides a reference-free light field image quality evaluation method based on a visual transformer with stronger generalization ability and more attention to angle features.

[0007] The present invention is achieved by at least one of the following technical solutions.

[0008] A method for evaluating the quality of a reference-free light field image based on a visual transformer comprises the following steps:

[0009] S1, obtaining a sub-aperture image array of a distorted light field image;

[0010] S2, extracting the sub-aperture image array of the distorted light field image, and respectively extracting a central sub-aperture image and a sub-aperture image array with a central angular resolution of a×a;

[0011] S3, inputting the central sub-aperture image of the distorted light field image into a spatial blocking module to obtain a spatial block of the distorted light field image rich in spatial information, and inputting the sub-aperture image array with a central angular resolution of a×a of the distorted light field image into an angular blocking module to obtain an angular block of the distorted light field image rich in angular information;

[0012] S4, using a spatial feature extraction network based on a visual transformer to obtain multi-layer features in the spatial domain of the distorted light field image on the spatial block of the distorted light field image, and using an angle feature extraction network based on a visual transformer to obtain multi-layer features in the angle domain of the distorted light field image on the angle block of the distorted light field image;

[0013] S5, splicing the multi-layer features in the spatial domain of the distorted light field image and the multi-layer features in the angular domain of the distorted light field image to obtain a fusion feature rich in spatial features and angular features;

[0014] S6. Use the quality score prediction network based on linear regression to regress the extracted fusion features to obtain the quality score of the distorted light field image, and use gradient back propagation to train the network according to the configured relevant network model and training parameters. Use the trained network to simulate the human eye's perception of the distorted light field image quality and evaluate the quality of the distorted light field image.

[0015] Furthermore, in step S1, the acquired sub-aperture image array is obtained by fixing the angular dimension of the distorted light field image; the distorted light field image originates from the light field image quality evaluation dataset Win5-LID, and a 9×9 image array is obtained by fixing the angular dimension; the sub-aperture image array image is represented by a dual-plane parameter method as l(u, v, h, w), where l represents light intensity, u and v represent angular resolution, and h and w represent length and width resolution of the space; for the WIN5-LIB dataset, the angular resolution of its sub-aperture image array is 9×9, that is, the sub-aperture image array of its distorted light field image is represented as L=l(9,9,h,w).

[0016] Further, in step S2, the central sub-aperture image is a central sub-aperture image extracted from a 9×9 sub-aperture image array, and the central sub-aperture image array is a central 7×7 sub-aperture image array extracted from a 9×9 sub-aperture image array.

[0017] Furthermore, in step S3, the spatial blocking module randomly crops the central sub-aperture image to obtain a distorted light field image spatial block of a size of 224×224, and maintains the RGB color space during the cropping process.

[0018] Furthermore, in step S3, the angle blocking module randomly crops the same position of each image in the 7×7 central sub-aperture image array to obtain an image block of size 32×32, and sequentially splices the cropped 32×32 image blocks into 224×224 distorted light field image angle blocks according to the position relationship of the 7×7 central sub-aperture image array, and maintains the RGB color space during the cropping process.

[0019] Furthermore, the spatial feature extraction network based on the visual transformer includes a visual transformer backbone network and a residual spatial network; the angle feature extraction network based on the visual transformer includes a visual transformer backbone network and a residual angle network;

[0020] The visual transformer backbone network includes an embedding layer, a position encoding layer and a plurality of transformer encoding layers;

[0021] The embedding layer converts the input into a sequence, divides the RGB input of size 224×224 into n image blocks of size 16×16, and processes the three dimensions of the color space into one dimension to obtain n tokens, each of which has a dimension of 768. Add category tokens of the same dimension to obtain a token sequence Z = [z class ;z1;z2;...;z n ], where z class is the category token of the same dimension, z n is the token processed by the three dimensions of the color space, where n represents the number of tokens.

[0022] The position encoding layer uses one-dimensional linear encoding to add position information to n tokens, allowing the network to learn the position relationship between tokens, that is, Z0 = [z class ;z1;z2;...;z n ]+E pos ,in Z0 is Z plus position information, E pos Indicates the location information between each token;

[0023] The transformer coding layer includes multi-head attention MSA, multi-layer perceptron MLP, two layer normalization LN and two residual connections; the calculation formula after multiple transformer coding layers are serially spliced ​​is:

[0024] Z′ l =MSA(LN(Z l-1 ))+Z l-1 , Z l =MLP(LN(Z′ l ))+Z′ l

[0025] Where l = 1...m, m = 12, Z′ l represents the intermediate features of the transformer encoding layer, Z l-1 represents the input of the lth layer, Z l represents the output of the lth layer, MSA represents multi-head attention, MLP represents multi-layer perceptron, LN represents layer normalization, and L is the number of transformer encoding layers.

[0026] The residual space network includes a convolutional layer with a size of 1×1, a stride of 1, and an output channel of 32, two convolutional layers with a size of 3×3, a stride of 1, and an output channel equal to the input channel, an LReLU layer, and a residual connection module;

[0027] The residual angle network includes a convolutional layer with a size of 1×1, a stride of 1, and an output channel of 32, three convolutional layers with a size of 3×3, a dilation rate of 1, 3, and 5, and an output channel of the input channel, an LReLU layer, and a residual connection module;

[0028] The residual connection module in the residual space network performs a residual connection on the output of the LReLU layer in the residual space network and the original input of the residual space network to generate a residual space network output;

[0029] The residual connection module in the angle space network performs a residual connection on the output of the LReLU layer in the residual angle network and the original input of the residual angle network to generate a residual angle network output.

[0030] Furthermore, in step S5, the fusion feature is to splice the multi-layer features in the spatial domain of the distorted light field image and the multi-layer features in the angle domain of the distorted light field image in the channel dimension.

[0031] Furthermore, the quality score prediction network includes a global average pooling layer and two fully connected layers, which regress the concatenated features to the quality scores of the distorted light field images.

[0032] Furthermore, step S6 is configured such that the initial learning rate of the training process is 0.00001, the learning rate is adjusted by using a cosine annealing function during the training process, the training batch is set to 16, and the number of training rounds is set to 100.

[0033] Furthermore, the loss function during training is shown in the following formula:

[0034]

[0035] Where L represents the loss value during training, q h represents the hth objective evaluation score, s hrepresents the hth subjective evaluation score, and H represents the total number of image blocks input into the network for evaluation.

[0036] Compared with the prior art, the present invention has the following advantages and effects:

[0037] The present invention designs a model based on a visual transformer, does not require a large amount of training data, and can achieve excellent performance on a small light field image quality evaluation data set. Random cropping of the central sub-aperture image in the spatial blocking module can achieve training data enhancement. The light field image angle block obtained by the angle blocking module can reduce the spatial resolution of the light field image, which can not only narrow the gap between the angle resolution and the spatial resolution, improve the importance of the angle feature, but also enhance the training data. In the spatial feature extraction network, the residual space network is connected after the visual transformer to make up for the defect that the visual transformer is difficult to collect fine-grained spatial features. In the angle feature extraction network, the residual angle network is connected after the visual transformer, and the hole convolution with different hole rates is used to further improve the extraction ability of the angle feature. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of a method for evaluating quality of a reference-free light field image based on a visual transformer according to an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of central sub-aperture image extraction provided by an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of 7×7 central sub-aperture image array extraction provided by an embodiment of the present invention;

[0041] Figure 4 A schematic diagram of specific operations of a spatial block module provided in an embodiment of the present invention;

[0042] Figure 5 A schematic diagram of the specific operation of the angle block module provided by an embodiment of the present invention;

[0043] Figure 6 A structural diagram of a spatial feature extraction network based on a visual transformer provided in an embodiment of the present invention;

[0044] Figure 7 A structural diagram of an angle feature extraction network based on a visual transformer provided in an embodiment of the present invention;

[0045] Figure 8 A framework diagram of a reference-free light field image quality assessment method based on a visual transformer provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0047] like Figure 1 As shown, a method for evaluating the quality of a reference-free light field image based on a visual transformer in this embodiment includes the following steps:

[0048] Step S1, obtaining a sub-aperture image array of a distorted light field image.

[0049] As one of the embodiments, this embodiment selects the public light field image quality evaluation dataset Win5-LID, which contains ten image contents and 220 distorted light field images, each of which has a subjective quality score. A 9×9 sub-aperture image array is obtained by fixing the angular dimension of the light field image in the dataset. The sub-aperture image array image can be expressed as l(u, v, h, w) using the dual-plane parameter method, where l represents light intensity, u and v represent angular resolution, and h and w represent spatial resolution.

[0050] For the WIN5-LIB dataset, the angular resolution of its subaperture image array is 9×9, that is, the subaperture image array of its distorted light field image can be expressed as l(u,v,h,w), u=1,2,...,9,v=1,2,...,9. When the angular dimension is fixed, the RGB color space of the light field image is not changed to avoid reintroducing distortion in the data preprocessing stage.

[0051] Step S2: extracting the sub-aperture image array of the distorted light field image, and respectively extracting a central sub-aperture image and a sub-aperture image array with a central angular resolution of 7×7.

[0052] The central sub-aperture image in step S2 is obtained by extracting the central sub-aperture image of a 9×9 sub-aperture image array. The specific operation is as shown in the attached figure. Figure 2 The central sub-aperture image array in step S2 is obtained by extracting the central 7×7 sub-aperture image array of the 9×9 sub-aperture image array. The specific operation is as shown in the attached Figure 3 shown.

[0053] Step S3: input the central sub-aperture image of the distorted light field image into a spatial blocking module to obtain a spatial block of the distorted light field image rich in spatial information, and input the sub-aperture image array with a central angular resolution of 7×7 of the distorted light field image into an angular blocking module to obtain an angular block of the distorted light field image rich in angular information.

[0054] The spatial block module in step S3 randomly crops the central sub-aperture image into spatial blocks of size 224×224. The specific operation is as shown in the attached figure. Figure 4 As shown. The angle block module described in step S3 randomly crops each sub-aperture image in the 7×7 sub-aperture image array at the same position and then splices them according to the position. After randomly cropping at the same position, 7×7 spatial blocks of size 32×32 are obtained, which are spliced ​​according to the position of the sub-aperture image in the sub-aperture image array to obtain an angle block of size 224×224. The specific operation is as shown in the attached figure. Figure 5 shown.

[0055] Each spatial block and angular block inherits the subjective quality score label of the original light field image. Through the spatial and angular block modules, the training data can be enhanced without introducing additional distortion and noise, the number of training samples can be increased, and the color information can be retained. Through the angular block module, the gap between angular resolution and spatial resolution can be narrowed, so that the subsequent network can pay more attention to the angular information in the light field image.

[0056] Step S4, using a spatial feature extraction network based on a visual transformer to obtain multi-layer features in the spatial domain of the distorted light field image on the spatial block of the distorted light field image, and using an angular feature extraction network based on a visual transformer to obtain multi-layer features in the angular domain of the distorted light field image on the angular block of the distorted light field image.

[0057] The spatial feature extraction network described in step S4 specifically includes a visual transformer backbone network and a residual spatial network, and the angle feature extraction network specifically includes a visual transformer backbone network and a residual angle network. The structure of the spatial feature extraction network is shown in the attached figure. Figure 6 As shown in the figure, the structure of the angle feature extraction network is shown in the attached figure. Figure 7 shown.

[0058] As one embodiment, the visual transformer backbone network consists of an embedding layer, a position encoding layer and twelve transformer encoding layers.

[0059] The embedding layer of the visual transformer backbone network can convert the input into a sequence. The RGB input of size 224×224 is divided into 196 image blocks of size 16×16, and the three dimensions of the color space are processed into one dimension to obtain 196 tokens, each with a dimension of 768. Adding the class token of the same dimension, we get the token sequence Z = [z class ;z1;z a ;...;z n ], where z class is the category token of the same dimension, zn It is the token processed above, where n represents the number of tokens, which is 196.

[0060] In the position encoding layer of the visual transformer backbone network, one-dimensional linear coding is used to add position information to the 197 tokens, so that the network can better distinguish the position relationship between tokens, that is, Z0 = [z class ;z1;z2;...;z n ]+E pos ,in Z0 is Z plus position information, E pos Represents the position information between each token. This is because the position information between the inputs will be lost after being converted into a sequence. The position encoding layer can more effectively introduce the position information between sequences.

[0061] The transformer coding layer in the visual transformer backbone network consists of a multi-head attention MSA, a multi-layer perceptron MLP, two layer normalization LNs and two residual connections. There are twelve transformer coding layers in the visual transformer backbone network, which extract the attention information of the spatial block and the angle block through serial splicing. The calculation formula after the twelve transformer coding layers are serially spliced ​​is: Z′ l =MSA(LN(Z l-1 ))+Z l-1 , Z l =MLP(LN(Z′ l ))+Z′ l , where l=1...m,m,=12. Z′ l represents the intermediate features of the transformer encoding layer, Z l-1 represents the input of the lth layer, Z l represents the output of the lth layer, MSA represents multi-head attention, MLP represents multi-layer perceptron, LN represents layer normalization, and L is the number of transformer encoding layers, which is set to 12.

[0062] The visual transformer backbone network in the spatial feature extraction network can perform attention calculations on spatial blocks, thereby simulating the human eye's attention to important information in the spatial domain. The visual transformer backbone network in the angle feature extraction network focuses on the attention calculations on angle blocks, simulating the human eye's attention to key information in the angle domain.

[0063] The residual spatial network in the spatial feature extraction network consists of a convolutional layer with a size of 1×1, a step size of 1, and an output channel of 32, two convolutional layers with a size of 3×3, a step size of 1, and an output channel as an input channel, an LReLU layer, and a residual connection module.

[0064] The residual angle network in the angle feature extraction network consists of a convolutional layer with a size of 1×1, a step size of 1, and an output channel of 32, three convolutional layers with a size of 3×3, a dilation rate of 1, 3, and 5, and an output channel of the input channel, an LReLU layer, and a residual connection module. The angle block passes through a convolutional layer with a size of 1×1, a step size of 1, and an output channel of 32, and is input into three dilated convolutional layers with different dilation rates. The three features output by the dilated convolution are spliced ​​according to the channel dimension.

[0065] The residual space network effectively captures fine-grained information through its design. The 1×1 convolution layer first compresses the number of channels and transforms the features of the input data, so that the subsequent 3×3 convolution layer can focus more on learning local features and maintain activation characteristics in each convolution layer. In particular, the introduction of the LReLU layer can alleviate the problem of gradient vanishing and enhance the ability of feature extraction. The use of the residual connection module further optimizes the flow of information, allowing the network to more effectively learn deeper features while avoiding performance degradation due to network depth.

[0066] The residual angle network makes full use of the characteristics of the dilated convolution to expand the receptive field and extract richer angle features. By introducing multiple 3×3 convolutional layers with different dilation rates, the network can capture features at different scales, thereby effectively integrating local and global information. The design of inputting the features of the 1×1 convolutional layer into the dilated convolutional layer also ensures full utilization of information, thereby realizing the fusion of multi-scale features. This feature splicing operation in the channel dimension can integrate the angle information under different receptive fields, making the final feature representation more comprehensive.

[0067] The residual connection module performs a residual connection on the aforementioned extracted features and the original input to generate a module output.

[0068] Step S5: splicing the multi-layer features of the distorted light field image in the spatial domain and the multi-layer features of the distorted light field image in the angular domain to obtain a fusion feature rich in spatial features and angular features.

[0069] The stitching described in step S5 is to stitch the multi-layer features in the spatial domain of the distorted light field image and the multi-layer features in the angle domain of the distorted light field image in the channel dimension.

[0070] Step S6: Use the quality score prediction network based on linear regression to regress the extracted fusion features to obtain the quality score of the distorted light field image, and use the gradient back propagation training network according to the configured relevant network model and training parameters. The overall network is as shown in the attached figure. Figure 8 shown.

[0071] The quality score prediction network described in step S6 consists of a global average pooling layer and two fully connected layers. The input of the quality score prediction network is a fused feature with a dimension of (64, 224, 224). The fused feature is subjected to a global average pooling layer to obtain a feature with a dimension of (64, 1). The pooled features can be regressed to the quality score of the distorted light field image through two fully connected layers. The configuration described in step S6 is that the initial learning rate of the training process is 0.00001, the cosine annealing function is used to adjust the learning rate during the training process, the training batch is set to 16, the number of training rounds is set to 100, and the loss function during the training process is shown in the following formula:

[0072]

[0073] Where L represents the loss value during training, q h represents the hth objective evaluation score, s h represents the hth subjective evaluation score, and H represents the total number of image blocks input into the network for evaluation.

[0074] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well.

Claims

1. A method for evaluating the quality of a reference-free light field image based on a visual transformer, characterized in that: The following steps are involved: S1, obtaining a sub-aperture image array of a distorted light field image; S2, extracting the sub-aperture image array of the distorted light field image, and respectively extracting a central sub-aperture image and a sub-aperture image array with a central angular resolution of a×a; S3, inputting the central sub-aperture image of the distorted light field image into a spatial blocking module to obtain a spatial block of the distorted light field image rich in spatial information, and inputting the sub-aperture image array with a central angular resolution of a×a of the distorted light field image into an angular blocking module to obtain an angular block of the distorted light field image rich in angular information; S4, using a spatial feature extraction network based on a visual transformer to obtain multi-layer features in the spatial domain of the distorted light field image on the spatial block of the distorted light field image, and using an angle feature extraction network based on a visual transformer to obtain multi-layer features in the angle domain of the distorted light field image on the angle block of the distorted light field image; S5, splicing the multi-layer features in the spatial domain of the distorted light field image and the multi-layer features in the angular domain of the distorted light field image to obtain a fusion feature rich in spatial features and angular features; S6. Use the quality score prediction network based on linear regression to regress the extracted fusion features to obtain the quality score of the distorted light field image, and use gradient back propagation to train the network according to the configured relevant network model and training parameters. Use the trained network to simulate the human eye's perception of the distorted light field image quality and evaluate the quality of the distorted light field image.

2. According to claim 1, a method for evaluating quality of a reference-free light field image based on a visual transformer is characterized in that: In step S1, the acquired sub-aperture image array is obtained by fixing the angular dimension of the distorted light field image; the distorted light field image originates from the light field image quality evaluation dataset Win5-LID, and a 9×9 image array is obtained by fixing the angular dimension; the sub-aperture image array image is represented by a dual-plane parameter method as l(u, v, h, w), where l represents light intensity, u and v represent angular resolution, and h and w represent length and width resolution of the space; for the WIN5-LIB dataset, the angular resolution of its sub-aperture image array is 9×9, that is, the sub-aperture image array of its distorted light field image is represented as L=l(9,9,h,w).

3. The method for evaluating the quality of a reference-free light field image based on a visual transformer according to claim 1, characterized in that: In step S2, the central sub-aperture image is a central sub-aperture image extracted from a 9×9 sub-aperture image array, and the central sub-aperture image array is a central 7×7 sub-aperture image array extracted from a 9×9 sub-aperture image array.

4. The method for evaluating quality of a reference-free light field image based on a visual transformer according to claim 1, characterized in that: In step S3, the spatial blocking module randomly crops the central sub-aperture image to obtain a distorted light field image spatial block of a size of 224×224, and maintains the RGB color space during the cropping process.

5. The method for evaluating the quality of a reference-free light field image based on a visual transformer according to claim 1, characterized in that: In step S3, the angle blocking module randomly crops the same position of each image in the 7×7 central sub-aperture image array to obtain an image block of size 32×32, and sequentially splices the cropped 32×32 image blocks into 224×224 distorted light field image angle blocks according to the position relationship of the 7×7 central sub-aperture image array, and maintains the RGB color space during the cropping process.

6. The method for evaluating quality of a reference-free light field image based on a visual transformer according to claim 1, characterized in that: The spatial feature extraction network based on the visual transformer includes a visual transformer backbone network and a residual spatial network; the angle feature extraction network based on the visual transformer includes a visual transformer backbone network and a residual angle network; The visual transformer backbone network includes an embedding layer, a position encoding layer and a plurality of transformer encoding layers; The embedding layer converts the input into a sequence, divides the RGB input of size 224×224 into n image blocks of size 16×16, and processes the three dimensions of the color space into one dimension to obtain n tokens, each of which has a dimension of 768. Add category tokens of the same dimension to obtain a token sequence Z = [z class ;z1;z2;...; z n ], where z class is the category token of the same dimension, z n is the token processed by the three dimensions of the color space, where n represents the number of tokens; The position encoding layer uses one-dimensional linear encoding to add position information to n tokens, allowing the network to learn the position relationship between tokens, that is, Z0 = [z class ;z1;z2;...;z n ]+E pos ,in Z0 is Z plus position information, E pos Indicates the location information between each token; The transformer coding layer includes multi-head attention MSA, multi-layer perceptron MLP, two layer normalization LN and two residual connections; the calculation formula after multiple transformer coding layers are serially spliced ​​is: WITH' l =MSA(LN(Z l-1 ))+Z l-1 ,WITH l =MLP(LN(Z′ l ))+Z′ l Where l = 1...m, m = 12, Z l ′ represents the intermediate features of the transformer encoding layer, Z l-1 represents the input of the lth layer, Z l represents the output of the lth layer, MSA represents multi-head attention, MLP represents multi-layer perceptron, LN represents layer normalization, and L is the number of transformer encoding layers; The residual space network includes a convolutional layer with a size of 1×1, a stride of 1, and an output channel of 32, two convolutional layers with a size of 3×3, a stride of 1, and an output channel equal to the input channel, an LReLU layer, and a residual connection module; The residual angle network includes a convolutional layer with a size of 1×1, a stride of 1, and an output channel of 32, three convolutional layers with a size of 3×3, a dilation rate of 1, 3, and 5, and an output channel of the input channel, an LReLU layer, and a residual connection module; The residual connection module in the residual space network performs a residual connection on the output of the LReLU layer in the residual space network and the original input of the residual space network to generate a residual space network output; The residual connection module in the angle space network performs a residual connection on the output of the LReLU layer in the residual angle network and the original input of the residual angle network to generate a residual angle network output.

7. The method for evaluating quality of a reference-free light field image based on a visual transformer according to claim 1, characterized in that: In step S5, the fusion feature is to splice the multi-layer features in the spatial domain of the distorted light field image and the multi-layer features in the angle domain of the distorted light field image in the channel dimension.

8. The method for evaluating quality of a reference-free light field image based on a visual transformer according to claim 1, characterized in that: The quality score prediction network consists of a global average pooling layer and two fully connected layers, which regress the concatenated features to the quality score of the distorted light field image.

9. The method for evaluating quality of a reference-free light field image based on a visual transformer according to claim 1, characterized in that: Step S6 is configured such that the initial learning rate of the training process is 0.00001, the cosine annealing function is used to adjust the learning rate during the training process, the training batch is set to 16, and the number of training rounds is set to 100.

10. The method for evaluating quality of a reference-free light field image based on a visual transformer according to claim 1, characterized in that: The loss function during training is shown in the following formula: Where L represents the loss value during training, q h represents the hth objective evaluation score, s h represents the hth subjective evaluation score, and H represents the total number of image blocks input into the network for evaluation.

Images