Stereo image quality evaluation method of multi-task coding-decoding device based on Transform

Abstract

The invention discloses a stereo image quality evaluation method of a multi-task coder-decoder based on Transform, and belongs to the technical field of image quality evaluation. The invention provides a multi-task coding-decoding device network architecture based on Transform. The multi-task coding-decoding device network architecture takes image depth estimation as an auxiliary task; a feedback mechanism of a human visual system can be better simulated by adopting a coder-decoder network, the problems existing in left and right view feature extraction are solved, and an information processing process from a global state to a local state is realized; in addition, the global interaction capability of Transform and the local attention capability of CNN are combined to enhance image features, depth estimation is introduced as an auxiliary task, depth prior information with higher quality can be provided, cross-view parallax fusion based on Transform is guided, the problem existing in left and right view feature fusion is solved, and more accurate binocular features are obtained. And feedback compensation is carried out on multi-level binocular feature fusion by using an attention mechanism to obtain a final pixel-level quality map, so that the image quality is accurately evaluated, a research scheme is provided for a deep learning method for non-reference stereo image quality evaluation, and the development of a visual task is effectively promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of image quality assessment, and in particular to a Transformer-based multi-task encoder-decoder stereo image quality assessment method. Background Art

[0002] In recent years, stereoscopic images have played a vital role in fields such as virtual reality, medical imaging, and autonomous driving. However, stereoscopic images will inevitably be distorted during the acquisition, transmission, encoding, and display process, which will affect the audience's visual experience and may even cause problems such as visual fatigue and motion sickness. Therefore, developing an objective, efficient, and accurate method for evaluating the quality of stereoscopic images is a hot topic in the field of computer vision research.

[0003] As a technology for specific applications, stereoscopic image quality evaluation technology has broad application prospects in many fields. In virtual reality and augmented reality applications, evaluating the quality of stereoscopic images can effectively enhance the user's sense of immersion and reduce visual fatigue; in the process of 3D movie and video production, quality evaluation methods can help creators monitor image quality in real time to ensure that the output stereoscopic video has good visual effects; in autonomous driving and drone vision systems, quality evaluation of stereoscopic images helps ensure that the system can correctly understand the surrounding three-dimensional environment and improve driving and navigation safety. Therefore, stereoscopic image quality evaluation technology not only has important academic research value, but also has huge market potential in practical applications.

[0004] Research on stereoscopic image quality evaluation technology can be traced back to the late 20th century. Early research mainly focused on subjective evaluation methods, that is, asking viewers to visually rate stereoscopic images. Although subjective evaluation of the perceived quality of stereoscopic images is authentic and reliable, it is time-consuming, laborious and difficult to promote on a large scale, and its results are often affected by individual differences among evaluators. Therefore, subjective evaluation methods are difficult to adapt to diverse application needs. With the development of computer technology, automatic and efficient objective evaluation methods have gradually become the focus of research.

[0005] Based on the dependence on reference images, objective stereo image quality evaluation methods can be divided into three categories: full reference (FR), reduced reference (RR), and no reference (NR). Because it is difficult to obtain high-quality reference images in real scenes, the applicability of full reference and semi-reference methods is limited. The no-reference stereo image quality evaluation method, on the other hand, performs quality assessment based only on the distorted stereo image itself without any reference images, which is more versatile and flexible in practical applications.

[0006] Traditional no-reference evaluation methods mainly rely on manually extracted features, such as edge information, color distribution, and texture complexity, and then use statistical methods or machine learning algorithms to evaluate quality. Although these methods can predict image quality to a certain extent, they usually cannot accurately capture the complex features in the image and show poor generalization. In recent years, the rise of deep learning technology has provided new ideas for no-reference stereo image quality evaluation technology. Since humans are one of the receptors of stereo image perception, the objective evaluation method of no-reference stereo image quality should be based on the formation process of human perception. The deep learning-based method extracts more complex image features through convolutional neural networks (CNN), realizes a more accurate modeling of the human visual system (HVS), and becomes one of the mainstream methods in the field of no-reference stereo image quality evaluation.

[0007] In view of the outstanding performance of CNN in computer vision tasks, researchers introduced CNN to the task of no-reference stereo image quality evaluation. CNN-based algorithms can not only automatically extract key features in images, but also achieve end-to-end overall parameter optimization. Since stereo images are composed of left-eye views and right-eye views, in order to enhance the accuracy of evaluation, such algorithms usually adopt a dual-branch or multi-branch network structure, learn the features of the left and right views at the same time, and then fuse the binocular features by connecting the features of the left and right views to obtain the final features for quality prediction. The overall structure can be divided into three parts: feature extraction, feature fusion and quality regression.

[0008] In 2022, Liu et al. proposed a two-stream interactive network, which extracts monocular features through left and right branches respectively. Asymmetric convolution kernels are introduced in both branches to enhance the model's ability to extract local features of the image. At the same time, the sum and difference information of the monocular features are used as the binocular features of the stereo image. After fusing the monocular features with the binocular features, the quality score is obtained by regression. In 2022, Hu et al. constructed a four-branch CNN evaluation method. This method extracts features from the left view, right view, binocular sum map and binocular difference map respectively, and enhances the visual feature expression ability by weighted splicing of the four features. Finally, the quality score of the image is obtained by regression through the fully connected layer. In 2023, Chang et al. proposed a coarse-to-fine feedback-guided network model that takes into account the characteristics of the dominant eye. The network contains two left and right branches, and three parallel dilated convolutions with different dilation rates in each branch extract low, medium and high spatial frequency information respectively. The binocular fusion features corresponding to the left and right views are then generated through a binocular fusion module based on bidirectional disparity. Finally, through the information feedback guidance module, binocular to monocular guidance is completed between different scales.

[0009] Although the above algorithms have achieved good evaluation performance, the CNN-based no-reference stereo image quality evaluation method usually faces the problem of how to better simulate the characteristics of the human visual system. Specifically, when extracting the features of the left and right views, the existing methods all adopt an encoder structure that extracts high-level features from low-level features, which is an information analysis process from local to global. In fact, when given a distorted image, human observers first perform global perception observations and then further complete complex local perceptions. Secondly, the disparity information between the left and right views plays a vital role in the process of human observers perceiving depth information and generating a sense of stereo when viewing stereo images. However, the existing methods mainly rely on methods such as connection, fusion, addition and subtraction to fuse the features of the left and right views, and do not explore the role of disparity information; although some methods consider disparity information, they ignore the depth information provided by stereo images.

[0010] In order to solve the above problems, the present invention proposes a Transformer-based multi-task encoder-decoder stereo image quality assessment method. Summary of the invention

[0011] The purpose of the present invention is to propose a Transformer-based multi-task encoder-decoder stereo image quality assessment method to solve the problems raised in the background technology.

[0012] In order to achieve the above object, the present invention adopts the following technical solutions:

[0013] The Transformer-based multi-task encoder-decoder stereo image quality assessment method includes the following steps:

[0014] S1. Design a multi-task framework for image quality assessment, wherein the multi-task framework consists of two encoder-decoder branches, one of which is used for image quality assessment of left and right images, and the other encoder-decoder module is used for depth estimation;

[0015] S2, given two input images F on the left and right l and F r , the encoder composed of VGG16 is used to extract multi-level monocular features, marked as Where i = 1, 2, 3;

[0016] S3, sending the multi-level features output by the encoder to the Global and Local Interaction Transformer (GLIT) module for enhancement;

[0017] S4, input the enhanced multi-level features into the decoder for upsampling and step-by-step fusion operations, and output the multi-level features marked as Where i = 1, 2, 3;

[0018] S5. For the decoder in the encoder-decoder branch for image quality assessment, a depth information guided Cross-view Parallax Fusion Transformer (DCPFT) module is designed to fuse left and right monocular features of different levels respectively.

[0019] S6. Design a Multi-Level Feedback Fusion (MLFF) module to input the three-level binocular features processed and output by the depth-guided cross-view disparity fusion transformer (DCPFT) module into the MLFF module to achieve feedback fusion.

[0020] S7. The binocular features at different levels are fused by the multi-level feature feedback fusion (MLFF) module to output the final quality feature F. Further upsampling operation is performed to obtain a pixel-level quality map. The sum of the quality scores of all pixels is calculated to obtain the final quality score.

[0021] Preferably, the global and local interactive transformer (GLIT) module in S3 is composed of two components: global and local content interactive attention (Global and Local Content Interaction Attention, GLCIA) and gated feed-forward network (GatedFeed-Forward Network, GFFN), and a jump connection is used between the two components; taking the global and local interactive transformer module processing process on the encoder-decoder branch for left view quality assessment as an example, its implementation process is expressed as follows:

[0022]

[0023] Among them, f GLCLA represents the processing operations of the global and local content interaction attention components; f GFFN represents the processing operation of the gated feedforward network component; f ln Represents layer normalization operation; ⊕ represents pixel-by-pixel addition; represents the image features obtained after layer normalization, where i = 1, 2, 3; represents the image features obtained after global and local content interactive attention and skip connection processing, where i = 1, 2, 3; Represents the graph features obtained after being enhanced by the global and local interactive transformer modules.

[0024] Preferably, the processing operation of the global and local content interaction attention (GLCIA) component specifically includes the following contents:

[0025] For input features Grouping along the channel dimension results in two sets of features Among them, H, W, and C represent the height, width, and number of channels respectively. and Enter into the global context branch and the local context branch respectively:

[0026] In the global context branch, the efficient self-attention mechanism (ESA) is used instead of the conventional self-attention mechanism. Through three convolution layers consisting of 1×1 convolution and 3×3 depth convolution, we get the matrix After transposing, we get and right Softmax function is performed along the column, and then Batch-wise matrix multiplication is performed between them, and the features obtained are processed by the softmax function along the row. Perform batch-wise matrix multiplication and finally pass through a 1×1 convolutional layer to obtain the final global context feature The implementation process is expressed as:

[0027]

[0028] in, and They are the softmax functions along the columns and rows respectively; represents Batch-wise matrix multiplication; f conv1 Indicates a convolution operation with a convolution kernel of 1×1;

[0029] In the local context branch, convolutional layers are used in parallel to extract local context features. Enhanced features are obtained through a residual block consisting of 1×1 convolution and 3×3 depth convolution. Then, the weights are learned through global average pooling (GAP), multi-layer perceptron (MLP) and sigmoid activation function to perform self-correction and obtain local context features. The implementation process is expressed as:

[0030]

[0031]

[0032] Among them, f conv1-dwconv3-conv1 represents a convolutional layer operation consisting of a 1×1 convolution, a 3×3 depthwise convolution, and a 1×1 convolution; f sigmoid-MLP-GAP Represents global maximum pooling, multi-layer perceptron and sigmoid activation function operations; ⊙ represents pixel-by-pixel multiplication;

[0033] Finally, the global context feature and local context features Merge to obtain richer image features after global and local content interactive attention processing The implementation process is expressed as:

[0034]

[0035] Among them, f concat Represents a concatenation operation along the channel dimension.

[0036] Preferably, the processing operation of the gated feedforward network (GFFN) component specifically includes the following contents:

[0037] The first fully connected layer and nonlinear layer in the ordinary feedforward network are improved, and a gated linear unit constructed by two linear projections (one of which is activated by the sigmoid function) is used to control the forward flow of feature information, so that the network focuses on features with richer information and improves module performance. The implementation process is expressed as follows:

[0038]

[0039] Among them, δ represents the sigmoid function; f dwconv3 Indicates a depthwise convolution operation with a convolution kernel of 3×3.

[0040] Preferably, the depth information guided cross-view parallax fusion transformer (DCPFT) module is composed of a depth information guided cross-view parallax fusion attention (DCPFA) component and a gated feed-forward network (GFFN) component, which processes the left and right view features output by the decoder. (i=1,2,3) In the fusion process, the image depth information obtained by the auxiliary task is aggregated (i=1,2,3), the implementation process is expressed as:

[0041]

[0042]

[0043] Among them, f DCPFA represents the operation of the cross-view disparity fusion attention component guided by depth information; f GFFN represents the operation of a gated feedforward network component; Represents the left and right view features obtained by cross-view disparity fusion attention processing guided by depth information.

[0044] Preferably, the implementation process of the depth information guided cross-view parallax fusion attention (DCPFA) component specifically includes the following contents:

[0045] Monocular context features extracted by the left and right branches in the image quality assessment task After the convolution layer consisting of 1×1 convolution and 3×3 depth convolution, the matrices are obtained respectively. and In order to obtain cross-view disparity information based on the left / right view, the matrix And the depth information obtained by the depth estimation auxiliary task Add together to get the query matrix with deep prior information The matrix And the depth information obtained by the depth estimation auxiliary task Add together to get the query matrix with deep prior information

[0046] Then the key matrix and Transpose and get the matrix In the matrix and matrix Batch-wise matrix multiplication is performed between them, and then the softmax function is used along the epipolar line to obtain the disparity attention map across views. The implementation process is expressed as:

[0047]

[0048]

[0049]

[0050]

[0051] Among them, f softmax represents the softmax function along the epipolar lines.

[0052] The value matrix Respectively with the obtained disparity attention map Ml→r / M r→l Perform batch-wise matrix multiplication and process it through a 1×1 convolution layer to obtain cross-view parallax compensation information Therefore, the binocular disparity feature Obtained by the following process:

[0053]

[0054]

[0055] Finally, the binocular disparity feature Aggregate and send to the gated feedforward network (GFFN) component to obtain the binocular features of cross-view disparity fusion

[0056]

[0057] Preferably, the multi-level feature feedback fusion (MLFF) module is used to realize the layer-by-layer fusion of binocular features at adjacent levels, and its implementation process is expressed as follows:

[0058] Given adjacent binocular features and As the input features of the multi-level feature feedback fusion module, first, Upsampling to get features To match the next level of features Resolution and channel dimensions;

[0059] Then, a Grouped Attention Gate (GAG) module is introduced to fuse and The Group Attention Gating (GAG) module uses the gating signal generated by high-level features to modulate lower-level features. and A convolution operation with a convolution kernel size of 3×3 is performed, and then batch normalization (BN) is used to standardize and add and merge; the generated feature map is activated by the Relu function, and then processed by 1×1 convolution and batch normalization layers to obtain a single-channel feature map. The single-channel feature map generates an attention coefficient through the sigmoid function. Modulated by the attention coefficient and then combined with the feature Add pixel by pixel to get fusion features Similarly, features After upsampling, the features are obtained and After the same steps as above, the final image quality feature F is obtained, and its implementation process is expressed as:

[0060]

[0061]

[0062]

[0063]

[0064] Among them, f GAG () indicates the grouped attention gating module operation; f gconv3 represents a group convolution with a kernel size of 3×3; f BN represents batch normalization operation; f BN-conv1-Relu Relu represents Relu function activation, 1×1 convolution operation and batch normalization layer processing; δ represents the sigmoid function.

[0065] Compared with the prior art, the present invention provides a Transformer-based multi-task encoder-decoder stereo image quality assessment method, which has the following beneficial effects:

[0066] The present invention aims to accurately evaluate the quality of stereo images and proposes a Transformer-based multi-task encoder-decoder network architecture with image depth estimation as an auxiliary task. The encoder-decoder network used in the present invention can better simulate the feedback mechanism of the human visual system, solve the problems existing in the feature extraction of left and right views, and realize the information processing process from global to local.

[0067] In addition, in order to prevent the semantic difference of image features between the encoder and the decoder, the Transformer-based multi-task encoder-decoder stereo image quality assessment method proposed in the present invention combines the global interaction capability of Transformer and the local attention capability of CNN to enhance image features, and introduces depth estimation as an auxiliary task, which can provide higher quality depth prior information, guide the Transformer-based cross-view disparity fusion, solve the problem of left and right view feature fusion, and obtain more accurate binocular features;

[0068] The Transformer-based multi-task encoder-decoder stereo image quality assessment method proposed in the present invention also uses the attention mechanism to perform feedback compensation on the multi-level binocular feature fusion to obtain the final pixel-level quality map, thereby accurately evaluating the image quality. This method provides a research plan for the deep learning method of reference-free stereo image quality evaluation, and effectively promotes the development of visual tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is an overall structural diagram of the Transformer-based multi-task encoder-decoder stereo image quality assessment method mentioned in Embodiment 1 of the present invention;

[0070] Figure 2 It is a specific structural diagram of the global and local interaction transformer (GLIT) module mentioned in Example 1 of the present invention;

[0071] Figure 3 This is a specific structural diagram of the depth information guided cross-view parallax fusion transformer (DCPFT) module mentioned in Example 1 of the present invention;

[0072] Figure 4 This is a specific structural diagram of the Multi-Level Feedback Fusion (MLFF) module just mentioned in Example 1 of the present invention. DETAILED DESCRIPTION

[0073] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0074] Embodiment 1:

[0075] See also Figure 1 , the present invention proposes a Transformer-based multi-task encoder-decoder stereo image quality assessment method, such as Figure 1 As shown, including the following:

[0076] This paper adopts a multi-task framework to evaluate image quality. The framework consists of two encoder-decoder branches, one for image quality evaluation of left and right images and the other for depth estimation. Given two input images, the encoder composed of VGG16 is first used to extract multi-level monocular features, marked as Where i = 1, 2, 3; the multi-level features output by the encoder are sent to the proposed global and local interaction transformer (GLIT) module for enhancement, and then input to the decoder for upsampling and step-by-step fusion operations, and the output multi-level features are marked as Where i = 1, 2, 3. In the decoder of the quality evaluation branch, a depth information guided Cross-view Parallax Fusion Transformer (DCPFT) module is designed to fuse the left and right monocular features of different levels respectively. The three levels of binocular features output by DCPFT are input into a Multi-Level Feedback Fusion (MLFF) module to realize feedback compensation connection. For quality regression, such as Figure 1 As shown in the figure, the binocular features at different levels are fused into the final quality features through MLFF, and a pixel-level quality map is obtained through upsampling. The sum of the quality scores of all pixels is calculated to obtain the final perceptual quality score. The details are described as follows.

[0077] See also Figure 2 The specific structure of GLIT is as follows: Figure 2 As shown in Figure 2. Similar to the structure of other transformers, GLIT consists of two core components: Global and Local Content Interaction Attention (GLCIA) and Gated Feed-Forward Network (GFFN), and both components use jump connections. Taking GLIT on the left view quality evaluation branch as an example, the entire process is defined as:

[0078]

[0079]

[0080]

[0081] Among them, f GLCLA represents the processing operations of the global and local content interaction attention components; f GFFN represents the processing operation of the gated feedforward network component; f ln Represents layer normalization operation; ⊕ represents pixel-by-pixel addition; represents the image features obtained after layer normalization, where i = 1, 2, 3; represents the image features obtained after global and local content interactive attention and skip connection processing, where i = 1, 2, 3; Represents the graph features obtained after being enhanced by the global and local interactive transformer modules.

[0082] For GLCIA, we first perform the input feature Grouping along the channel dimension results in two sets of features Among them, H, W, and C represent the height, width, and number of channels respectively. and Enter into the global context branch and the local context branch respectively.

[0083] In the global context branch, since the self-attention mechanism causes the computational cost to grow quadratically with the spatial resolution of the feature map, this module uses the Efficient Self-Attention (ESA) mechanism instead, which has equivalent effects but only linear computational cost. Through three convolution layers consisting of 1×1 convolution and 3×3 depth convolution, we get the matrix After transposing, we get and right Softmax function is performed along the column, and then Batch-wise matrix multiplication is performed between them, and the features obtained are processed by the softmax function along the rows and columns. Perform batch-wise matrix multiplication and finally pass through a 1×1 convolutional layer to obtain the final global context feature The implementation process can be expressed as:

[0084]

[0085] in, and They are the softmax functions along the columns and rows respectively; represents Batch-wise matrix multiplication; f conv1 Indicates a convolution operation with a convolution kernel of 1×1.

[0086] In the local context branch, considering that the convolution operation is better at capturing local features in the image, this method uses convolution layers in parallel to extract local context features. Enhanced features are obtained through a residual block consisting of 1×1 convolution and 3×3 depth convolution. Then, the weights are learned through global average pooling (GAP), multi-layer perceptron (MLP) and sigmoid activation function to perform self-correction and obtain local context features.

[0087]

[0088]

[0089] Among them, f conv1-dwconv3-conv1 represents a convolutional layer operation consisting of a 1×1 convolution, a 3×3 depthwise separable convolution, and a 1×1 convolution; f sigmoid-MLP-GAP represents the global maximum pooling, multi-layer perceptron and sigmoid activation function operations; ⊙ represents pixel-by-pixel multiplication.

[0090] Finally, the global context feature and local context features Merge to obtain richer image features after global and local content interactive attention processing The implementation process is expressed as:

[0091]

[0092] Among them, f concat Represents a concatenation operation along the channel dimension.

[0093] For the GFFN module, the first fully connected layer and nonlinear layer in the ordinary feedforward network are improved, and a gated linear unit constructed by two linear projections (one of which is activated by the sigmoid function) is used to control the forward flow of feature information, so that the network focuses on features with richer information and improves module performance. This process is implemented by the following formula:

[0094]

[0095] Among them, δ represents the sigmoid function; f dwconv3 Indicates a depthwise convolution operation with a convolution kernel of 3×3.

[0096] In summary, GLIT combines the characteristics of Transformer and CNN, extracts the global and local features of the image in parallel, obtains higher quality features while reducing computational complexity.

[0097] In order to obtain accurate binocular features, the method of the present invention innovatively designs a depth information guided cross-view parallax fusion transformer (DCPFT) to process the left and right view features output by the decoder. (i=1,2,3) In the fusion process, the image depth information is aggregated (i=1,2,3). It should be noted that the depth information It is extracted by the auxiliary task depth estimation network and can provide important geometric features to enhance the disparity correspondence between the left and right view features, thereby guiding the acquisition of more accurate binocular features.

[0098] The structure of DCPFT is as follows Figure 3As shown in Figure 1, DCPFT also uses two core components: Depth guided Cross-view Parallax Fusion Attention (DCPFA) and Gated Feedforward Network GFFN. The overall process is defined as:

[0099]

[0100]

[0101] Among them, f DCPFA represents the operation of the cross-view disparity fusion attention component guided by depth information; f GFFN represents the operation of a gated feedforward network component; Represents the left and right view features obtained by cross-view disparity fusion attention processing guided by depth information.

[0102] See also Figure 3 , the structure of DCPFA is as follows Figure 3 As shown in Figure 2. Monocular context features extracted by the left and right branches in the image quality assessment task After the convolution layer consisting of 1×1 convolution and 3×3 depth convolution, the matrices are obtained respectively. and In order to obtain the cross-view disparity information based on the left / right view, we first And the depth information obtained by the depth estimation auxiliary task Add together to get the query matrix with deep prior information Then the key matrix and After transposing, we get In the matrix and Batch-wise matrix multiplication is performed between them, and then the softmax function is used along the epipolar line to obtain the disparity attention map across views. This process is represented as follows:

[0103]

[0104]

[0105]

[0106]

[0107] Among them, f softmax represents the softmax function along the epipolar lines.

[0108] The value matrix Respectively with the obtained disparity attention map M l→r / M r→l Perform batch-wise matrix multiplication and process it through a 1×1 convolution layer to obtain cross-view parallax compensation information Therefore, the binocular disparity feature Obtained by the following process:

[0109]

[0110]

[0111] Finally, the binocular disparity feature Aggregate and send to GFFN to obtain binocular features of cross-view disparity fusion

[0112]

[0113] Inspired by the hierarchical fusion processing and top-down feedback mechanism of the human visual cortex, this method designs a multi-level feedback fusion MLFF module to achieve layer-by-layer fusion of binocular features at adjacent levels. The detailed structure of MLFF is shown in Figure 4 As shown, given the adjacent binocular features and As the input feature of the MLFF module. First, Upsampling to get features To match the next level of features Then, a Grouped Attention Gate (GAG) module is introduced to fuse and This module uses the gating signals generated by high-level features to modulate lower-level features to improve their accuracy.

[0114] Specifically, we first analyze the features and A convolution operation with a convolution kernel size of 3×3 is performed, and then batch normalization (BN) is used to standardize and add and merge. The generated feature map is activated by the Relu function, and then processed by 1×1 convolution and BN layer to obtain a single-channel feature map. The single-channel feature map generates the attention coefficient through the sigmoid function. Modulated by the attention coefficient and then combined with the feature Add pixel by pixel to get fusion features Similarly, features After upsampling, the features are obtained and After the same steps as above, the final image quality feature F is obtained. The implementation process is as follows:

[0115]

[0116]

[0117]

[0118]

[0119] Among them, f GAG () indicates the grouped attention gating module operation; f gconv3 represents a group convolution with a kernel size of 3×3; f BN represents batch normalization operation; f BN-conv1-Relu Relu represents Relu function activation, 1×1 convolution operation and batch normalization layer processing; δ represents the sigmoid function.

[0120] Embodiment 2:

[0121] Based on Example 1, but different in that, the Transformer-based multi-task encoder-decoder stereo image quality assessment method proposed in the present invention is explained below with a specific example, and the specific content is as follows.

[0122] 1. Benchmark Datasets and Evaluation Metrics

[0123] The proposed Transformer-based multi-task encoder-decoder stereo image quality assessment method is experimentally verified on four benchmark datasets. The datasets used include:

[0124] (1) LIVE 3D Phase I database: referred to as LIVE I database, contains 20 reference stereo images and 365 distorted stereo images. All distorted stereo images are symmetrically distorted, including five types of distortion: fast fading (FF), white Gaussian noise (WN), JPEG compression (JPEG), JPEG2000 compression (JP2K) and Gaussian blur (Gblur). The Differential Mean Opinion Score (DMOS) value is the perceptual quality label of the distorted stereo images.

[0125] (2) LIVE 3D Phase II database: referred to as LIVE II database, it contains 8 reference stereo images and 360 distorted stereo images, including both symmetrical and asymmetrical distorted stereo images, and 5 distortion types: FF, WN, JPEG, JP2K, and Gblur. The DMOS value is the perceptual quality label of the distorted stereo images.

[0126] (3) Waterloo-IVC 3D Phase I database: abbreviated as WIVC I database, it contains 6 reference stereo images and 330 distorted stereo images, including symmetrical and asymmetrical distorted stereo images. The distortion types include WN, Gblur and JPEG. The mean subjective score (MOS) value is the perceptual quality label of the distorted stereo images.

[0127] (4) Waterloo-IVC 3D Phase II database: abbreviated as WIVC II database, it contains 10 reference stereo images and 460 distorted stereo images. It also contains stereo images with symmetric distortion types and asymmetric distortion types. The distortion types included are WN, Gblur and JPEG. The MOS value is the perceptual quality label of the distorted stereo images.

[0128] The present invention uses three commonly used evaluation indicators for performance evaluation:

[0129] Pearson Linear Correlation Coefficient (PLCC), Spearman Rank Order Correlation Coefficient (SROCC) and Root Mean Square Error (RMSE). Among them, PLCC represents the linear dependence between objective scores and subjective scores, SROCC reflects the monotonicity of the prediction, and RMSE reflects the accuracy of the prediction. For PLCC and SROCC, the larger their values, the better the performance, while the trend of RMSE is the opposite.

[0130] 2. Training Settings

[0131] The present invention adopts a block-based training strategy to achieve data augmentation. First, the stereo image is divided into non-overlapping image blocks of 40×40 size, and then these non-overlapping image blocks are sent to the network for training. In the training stage, the subjective score (i.e., DMOS / MOS value) and depth map of the stereo image in the database are used as the label of each image block. This experiment uses the Pytorch framework to train the proposed model. The initial learning rate is set to 0.001, the batch size is set to 64, and the Euclidean loss and Stochastic Gradient Descent (SGD) optimizer are used to adjust the network parameters.

[0132] 3. Experimental results

[0133] Table 1 Experimental results on the benchmark datasets LIVE I and LIVE II. The best performance is marked in bold.

[0134]

[0135]

[0136] Table 2 Experimental results on the benchmark datasets WIVC I and WIVC II. The best performance is marked in bold.

[0137]

[0138] Tables 1 and 2 show the quantitative evaluation results of the method proposed in the present invention and some of the latest stereo image quality evaluation methods on four data sets, and the optimal indicators are marked in bold. As can be seen from Tables 1 and 2, the method of the present invention has achieved the highest accuracy on all four data sets, proving the excellent performance of the method of the present invention. This is due to the fact that the method of the present invention adopts an encoder-decoder network, which better simulates the feedback mechanism of the human visual system and realizes the information processing process from global to local during monocular feature extraction. At the same time, the global interaction ability of Transformer and the local attention ability of CNN are combined to enhance the image features, preventing the semantic differences of image features between the encoder and decoder from being misaligned. In addition, the method of the present invention introduces depth estimation as an auxiliary task, provides depth prior information, guides cross-view disparity fusion, and obtains more accurate binocular features. The method also performs feedback compensation fusion on multi-level binocular features to generate a pixel-level quality map, thereby more accurately evaluating image quality.

[0139] Table 3 SROCC indicators of single distortion type on benchmark datasets LIVE I and LIVE II. The best performance is marked in bold.

[0140]

[0141]

[0142] Table 4 PLCC indicators for single distortion type on benchmark datasets LIVE I and LIVE II. The best performance is marked in bold.

[0143] Tables 3 and 4 provide the comparison results of the proposed method with some of the latest stereo image quality evaluation methods on LIVEI and LIVE II for different distortion types, and the best index is marked in bold. This better demonstrates the accuracy of the proposed method in predicting the stereo image quality degradation caused by different distortion types.

[0144] Table 5 Ablation experiment results of network architecture, the best performance is bolded

[0145]

[0146] Table 6 Ablation experiment results of the main module, the best performance is bolded

[0147]

[0148] Tables 5 and 6 perform ablation experiments on the overall network framework of the method of the present invention and the role of each module. The experimental results in Table 5 fully demonstrate the positive effects of the introduction of the encoding-decoding structure and multi-task on the overall network. Table 6 proves the effectiveness of each main module for the method.

[0149] It should be noted that, in the present invention patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0150] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A Transformer-based multi-task encoder-decoder stereo image quality assessment method, characterized in that: The following steps are involved: S1. Design a multi-task framework for image quality assessment, wherein the multi-task framework consists of two encoder-decoder branches, one of which is used for image quality assessment of left and right images, and the other encoder-decoder module is used for depth estimation; S2, given two input images F on the left and right l and F r , the encoder composed of VGG16 is used to extract multi-level monocular features, marked as Where i = 1, 2, 3; S3, sending the multi-level features output by the encoder to the global and local interactive transformer modules for enhancement; S4, input the enhanced multi-level features into the decoder for upsampling and step-by-step fusion operations, and output the multi-level features marked as Where i = 1, 2, 3; S5. For the decoder in the encoder-decoder branch for image quality assessment, a depth-guided cross-view disparity fusion transformer module is designed to fuse left and right monocular features of different levels respectively. S6. Design a multi-level feature feedback fusion module, and input the three-level binocular features processed and output by the cross-view disparity fusion transformer module guided by depth information into the multi-level feature feedback fusion module to realize feedback compensation connection; S7. The binocular features at different levels are fused by the multi-level feature feedback fusion module to output the final quality feature F. Further upsampling operation is performed to obtain a pixel-level quality map. The sum of the quality scores of all pixels is calculated to obtain the final quality score.

2. The Transformer-based multi-task encoder-decoder stereo image quality assessment method according to claim 1, characterized in that: The global and local interactive transformer module in S3 is composed of two components: global and local content interactive attention and gated feedforward network, and a jump connection is used between the two components; taking the global and local interactive transformer module processing process on the encoder-decoder branch for left view quality assessment as an example, its implementation process is expressed as follows: Among them, f GLCLA represents the processing operations of the global and local content interaction attention components; f GFFN represents the processing operation of the gated feedforward network component; f ln Representation layer normalization operation; Indicates pixel-by-pixel addition; represents the image features obtained after layer normalization, where i = 1, 2, 3; represents the image features obtained after global and local content interactive attention and skip connection processing, where i = 1, 2, 3; Represents the graph features obtained after being enhanced by the global and local interactive transformer modules.

3. The Transformer-based multi-task encoder-decoder stereo image quality assessment method according to claim 2, characterized in that: The processing operations of the global and local content interaction attention components specifically include the following: For input features Grouping along the channel dimension results in two sets of features Among them, H, W, and C represent the height, width, and number of channels respectively; and Enter into the global context branch and the local context branch respectively: In the global context branch, an efficient self-attention mechanism is used instead of the conventional self-attention mechanism. Through three convolution layers consisting of 1×1 convolution and 3×3 depth convolution, we get the matrix After transposing, we get and right Softmax function is performed along the column, and then Batch-wise matrix multiplication is performed between them, and the features obtained are processed by the softmax function along the row. Perform batch-wise matrix multiplication and finally pass through a 1×1 convolutional layer to obtain the final global context feature The implementation process is expressed as: in, and They are the softmax functions along the columns and rows respectively; represents Batch-wise matrix multiplication; f conv1 Indicates a convolution operation with a convolution kernel of 1×1; In the local context branch, convolutional layers are used in parallel to extract local context features. Enhanced features are obtained through a residual block consisting of 1×1 convolution and 3×3 depth convolution. Then, the weights are learned through global maximum pooling, multi-layer perceptron and sigmoid activation function, and self-correction is performed to obtain local context features. The implementation process is expressed as: Among them, f conv1-dwconv3-conv1 represents a convolutional layer operation consisting of a 1×1 convolution, a 3×3 depthwise convolution, and a 1×1 convolution; f sigmoid-MLP-GAP Represents global maximum pooling, multi-layer perceptron and sigmoid activation function operations; ⊙ represents pixel-by-pixel multiplication; Finally, the global context feature and local context features Merge to obtain richer image features after global and local content interactive attention processing The implementation process is expressed as: Among them, f concat Represents a concatenation operation along the channel dimension.

4. The Transformer-based multi-task encoder-decoder stereo image quality assessment method according to claim 1, characterized in that: The processing operations of the gated feedforward network component specifically include the following: The first fully connected layer and nonlinear layer in the ordinary feedforward network are improved, and a gated linear unit constructed by two linear projections is used to control the forward flow of feature information, so that the network focuses on features with richer information and improves module performance. The implementation process is expressed as follows: Among them, δ represents the sigmoid function; f dwconv3 Indicates a depthwise convolution operation with a convolution kernel of 3×3.

5. The Transformer-based multi-task encoder-decoder stereo image quality assessment method according to claim 1, characterized in that: The depth information guided cross-view disparity fusion transformer module consists of a depth information guided cross-view disparity fusion attention component and a gated feedforward network component, which processes the left and right view features output by the decoder. In the fusion process, the image depth information obtained by the auxiliary task is aggregated The implementation process is expressed as: Among them, f DCPFA represents the operation of the cross-view disparity fusion attention component guided by depth information; f GFFN represents the operation of a gated feedforward network component; Represents the left and right view features obtained by cross-view disparity fusion attention processing guided by depth information.

6. The Transformer-based multi-task encoder-decoder stereo image quality assessment method according to claim 5, characterized in that: The implementation process of the depth information guided cross-view disparity fusion attention component specifically includes the following contents: Monocular context features extracted by the left and right branches in the image quality assessment task After the convolution layer consisting of 1×1 convolution and 3×3 depth convolution, the matrices are obtained respectively. and In order to obtain cross-view disparity information based on the left / right view, the matrix And the depth information obtained by the depth estimation auxiliary task Add together to get the query matrix with deep prior information The matrix And the depth information obtained by the depth estimation auxiliary task Add together to get the query matrix with deep prior information Then the key matrix and Transpose and get the matrix In the matrix and matrix Batch-wise matrix multiplication is performed between them, and then the softmax function is used along the epipolar line to obtain the disparity attention map across views. The implementation process is expressed as: Among them, f softmax represents the softmax function along the epipolar line; The value matrix Respectively with the obtained disparity attention map M l→r / M r→l Perform batch-wise matrix multiplication and process it through a 1×1 convolution layer to obtain cross-view parallax compensation information Therefore, the binocular disparity feature Obtained by the following process: Finally, the binocular disparity feature Aggregate and send to the gated feedforward network component to obtain binocular features of cross-view disparity fusion 7. The Transformer-based multi-task encoder-decoder stereo image quality assessment method according to claim 1, characterized in that: The multi-level feature feedback fusion module is used to realize the layer-by-layer fusion of binocular features at adjacent levels, and its implementation process is expressed as follows: Given adjacent binocular features and As the input features of the multi-level feature feedback fusion module, first, Upsampling to get features To match the next level of features resolution and channel dimension; then, a grouped attention gating module is introduced to fuse and The grouped attention gating module uses the gating signal generated by high-level features to modulate the lower-level features. and Perform a group convolution operation with a convolution kernel size of 3×3, and then use batch normalization to standardize and add and merge; After the generated feature map is activated by the Relu function, it is processed by 1×1 convolution and batch normalization layer to obtain a single-channel feature map. The single-channel feature map generates the attention coefficient through the sigmoid function. Modulated by the attention coefficient and then combined with the feature Add pixel by pixel to get fusion features Similarly, features After upsampling, the features are obtained and After the same steps as above, the final image quality feature F is obtained, and its implementation process is expressed as: Among them, f GAG () indicates the grouped attention gating module operation; f gconv3 represents a group convolution with a kernel size of 3×3; f BN represents batch normalization operation; f BN-conv1-Relu Relu represents Relu function activation, 1×1 convolution operation and batch normalization layer processing; δ represents the sigmoid function.

Images