Full-reference panoramic image quality assessment method and system based on block sequence similarity

Abstract

The present invention proposes a full-reference panoramic image quality assessment method and system based on block sequence similarity, the method comprising: obtaining image blocks containing rich semantic information in reference panoramic images and distorted panoramic images, extracting perceptual features of the image blocks using a dual-branch feature extractor consisting of a Swin Transformer and an equidistant deformable convolutional block, splicing the perceptual features of the reference image blocks and the distorted image blocks to form a reference image block sequence and a distorted image block sequence, measuring the similarity between the reference and distorted image block sequences through distortion-aware interaction, and performing regression operations on the obtained results using position attention calculation to obtain a final predicted quality score. The present invention has been extensively experimented in a non-uniformly distorted panoramic image quality assessment database, and has good robustness to non-uniform distortion.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and multimedia digital image processing, and in particular to a full-reference panoramic image quality evaluation method and system based on block sequence similarity. Background Art

[0002] Virtual reality (VR) is a computer simulation technology that can create and experience a virtual world. It has become increasingly common in all walks of life. VR has subverted the user's sensory experience and shaped a new interactive mode of "immersive communication". As a core part of VR technology, the visual quality of panoramic images (OI) is crucial to the experience of VR users. Through a head-mounted display (HMD), OI can provide VR users with a 360° immersive viewing field, allowing users to experience the virtual environment more deeply. However, OI will inevitably introduce some distortion during the process of shooting, transmission, compression and display, affecting the user's immersive experience. In order to evaluate and optimize the user's quality of experience (QoE), it is necessary to develop a method for panoramic image quality assessment (OIQA). OIQA can be divided into subjective OIQA and objective OIQA. The former evaluates image quality through the observer's subjective rating and the latter refers to the establishment of an accurate computational model consistent with human perception. Although subjective OIQA requires a lot of time and human resources, it provides a gold standard for benchmarking objective OIQA models. Compared with subjective OIQA, objective OIQA models are more friendly to real applications because they can be easily embedded in algorithm improvements and system optimization. In general, according to the degree of dependence of the objective algorithm on the reference image, the existing OI objective evaluation methods mainly include full reference OIQA (FR-OIQA) and no reference / blind OIQA (NR- / BOIQA). FR-OIQA requires the use of all reference image information when deployed, while NR- / BOIQA can evaluate OI quality without reference image information.

[0003] In recent years, many image quality assessment (IQA) methods have been proposed to establish a solid foundation for solving the OIQA problem. FR-IQA has made significant progress on 2D images, which are often difficult to be directly applied to OI due to its wide-angle field of view and spherical projection method, which makes it visually very different from 2D images. A simple way to design a model for FR-OIQA is to directly adopt FR-IQA methods by considering the spherical properties of OI or through a perceptually guided weighting scheme, but their performance is not ideal. Traditional FR-OIQA methods are based on traditional FR-IQA and improved by combining sphere representation and human preferences on different regions. Due to the complex interaction between users and visual content during browsing, their performance is far from satisfactory. To alleviate this dilemma, considering the fact that users can only see a limited field of view, a viewport-based FR-OIQA method is proposed, which extracts several viewports from OI and measures them through a FR-IQA model following a weighting strategy. In fact, this method has significant advantages over traditional FR-OIQA models.

[0004] However, they still have two disadvantages, namely, the extra computational load of format conversion and viewport generation if the user's scanning path does not exist; and the artificially weighted methods cannot simulate the user's viewing behavior well, especially in the face of non-uniform distortion. Summary of the invention

[0005] In view of the above situation, the main purpose of the present invention is to propose a full-reference panoramic image quality assessment method and system based on block sequence similarity to solve the above technical problems.

[0006] The present invention proposes a full-reference panoramic image quality assessment method based on block sequence similarity, the method comprising the following steps:

[0007] Step 1, obtaining a reference image block and a distorted image block rich in semantic information from the reference panoramic image and the distorted panoramic image;

[0008] A dual-branch feature extractor is constructed based on Swin Transformer and isometric deformable convolution blocks. The dual-branch feature extractor is used to extract perceptual features of the reference image block and the distorted image block respectively, so as to obtain the perceptual features of the reference image block and the distorted image block.

[0009] Step 2: splicing the perceptual features of the reference image block and the distorted image block to form a reference image block sequence and a distorted image block sequence, and measuring the similarity between the reference image block sequence and the distorted image block sequence through distortion perception interaction to obtain a similarity feature;

[0010] Step 3: Perform block position attention calculation on the similarity features to achieve information interaction of similarity features in the horizontal and vertical directions, enhance the global information expression ability in the feature map and the sensitivity of the model to spatial position information, and obtain the similarity features after information interaction;

[0011] Step 4: Perform a regression operation on the similarity features after information interaction to obtain the perceptual quality score of the non-uniformly distorted panoramic image.

[0012] The present invention also proposes a full-reference panoramic image quality assessment system based on block sequence similarity, wherein the system applies the full-reference panoramic image quality assessment method based on block sequence similarity as described above, and the system comprises:

[0013] Image block sampling module, used to:

[0014] Obtaining a reference image block and a distorted image block rich in semantic information from the reference panoramic image and the distorted panoramic image;

[0015] A dual-branch feature extractor is constructed based on Swin Transformer and isometric deformable convolution blocks. The dual-branch feature extractor is used to extract perceptual features of the reference image block and the distorted image block respectively to obtain the perceptual features of the reference image block and the distorted image block.

[0016] Distortion-aware interaction module for:

[0017] The perceptual features of the reference image block and the distorted image block are concatenated to form a reference image block sequence and a distorted image block sequence, and the similarity between the reference image block sequence and the distorted image block sequence is measured through distortion perception interaction to obtain a similarity feature;

[0018] Position attention module, used to:

[0019] Perform block position attention calculation on similarity features to achieve information interaction of similarity features in the horizontal and vertical directions, enhance the global information expression ability in the feature map and the model's sensitivity to spatial position information, and obtain similarity features after information interaction;

[0020] Quality regression module for:

[0021] The similarity features after information interaction are regressed to obtain the perceptual quality score of the non-uniformly distorted panoramic image.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1. The present invention directly performs block extraction on the original ERP image without viewport extraction, thereby avoiding additional computational burden and improving the versatility of the model.

[0024] 2. The present invention processes the irregular semantic features and two-level distortion problems of ERP images through a dual-branch feature extractor composed of Swin Transformer and isometric deformable convolutional blocks, thereby effectively extracting quality perception features.

[0025] 3. The present invention measures the perceptual distance between the reference image block sequence and the distorted image block sequence by constructing a perceptual distortion interaction module to measure the similarity between the two sequences. The module combines multi-head self-attention and cross-attention mechanisms to capture the potential correlation between the reference image block sequence and the distorted image block sequence.

[0026] 4. This invention effectively integrates the information interaction in the horizontal and vertical directions in the feature map by constructing a position attention module, and enhances the model's sensitivity to spatial position information. It not only retains the local information in the original features, but also enhances the correlation between features in a global range, significantly improving the model's expressiveness in complex visual tasks.

[0027] 5. The full-reference panoramic image quality assessment method based on block sequence similarity proposed in the present invention outperforms the existing FR-IQA method and FR-OIQA method, and has strong flexibility and adaptability. This flexibility and adaptability make the assessment model more widely applicable and practical, and can cope with various complex image quality assessment problems.

[0028] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A flowchart of a full-reference panoramic image quality assessment method based on block sequence similarity proposed by the present invention;

[0030] Figure 2 The overall framework of the full-reference panoramic image quality assessment method based on block sequence similarity proposed in the present invention;

[0031] Figure 3 This is a schematic diagram of the structure of the full-reference panoramic image quality assessment system based on block sequence similarity proposed by the present invention. DETAILED DESCRIPTION

[0032] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0033] These and other aspects of the embodiments of the present invention will be apparent with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0034] See also Figure 1 and Figure 2 , in the figure, , position coding; , splicing; , rectified linear unit; , batch normalization; , subtract corresponding elements; , element-wise multiplication; , add the corresponding elements.

[0035] This embodiment provides a full-reference panoramic image quality assessment method based on block sequence similarity, the method comprising the following steps:

[0036] Step 1, obtaining a reference image block and a distorted image block rich in semantic information from the reference panoramic image and the distorted panoramic image;

[0037] A dual-branch feature extractor is constructed based on Swin Transformer and isometric deformable convolution blocks. The dual-branch feature extractor is used to extract perceptual features of the reference image block and the distorted image block respectively, so as to obtain the perceptual features of the reference image block and the distorted image block.

[0038] In this step, a reference image block and a distorted image block rich in semantic information are obtained from the reference panoramic image and the distorted panoramic image, which specifically includes the following steps:

[0039] The sampling image block size is set, and the image sampling extracts an image block of 896×896 in size; to prevent overfitting, this embodiment uses 80% of the images in the non-uniform distortion panoramic image database for training, and 20% for testing.

[0040] In this step, a dual-branch feature extractor is used to extract perceptual features of the reference image block and the distorted image block respectively, so as to obtain perceptual features of the reference image block and the distorted image block, which specifically includes the following steps:

[0041] The Swin Transformer network and the first equidistant deformable convolution are used to construct the Swin Transformer branch. The perceptual features of the reference image block and the distorted image block are extracted through the Swin Transformer branch. Specifically, there is the following relationship:

[0042] ;

[0043] in, Indicates reference image blocks, Indicates Distorted image blocks, represents the parameters of the SwinTransformer pre-trained network, represents the Swin Transformer network, represents an isometric deformable convolution operation, Indicates the number of image blocks, which is set to 10 here. Indicates The perceptual features extracted by the SwinTransformer branch of the reference image blocks, Indicates The perceptual features extracted by the Swin Transformer branch of the distorted image block. The size of the reference perceptual features and the distorted perceptual features extracted by the Swin Transformer branch is .

[0044] The second equidistant deformable convolution is used to construct an equidistant deformable convolution block branch, and the perceptual features extracted from the reference image block and the distorted image block are extracted through the equidistant deformable convolution block branch. Specifically, the following relationship exists:

[0045] ;

[0046] in, represents the isometric deformable convolutional block branch, Indicates The perceptual features extracted by the equidistant deformable convolutional block branches of the reference image blocks are: Indicates The perceptual features of the distorted image patches are extracted by the equidistant deformable convolutional block branch.

[0047] The isometric deformable convolution block branch is composed of 6 isometric deformable convolution blocks stacked together, each of which contains a Convolution layer, an isometric deformable convolution layer, a normalization layer, a linear correction layer and a maximum pooling layer, specifically there is the following relationship:

[0048] ;

[0049] in, express Convolution operation, represents the linear correction operation, represents the normalization operation, represents the pooling operation, express Activation function. The size of the reference perception feature and distortion perception feature extracted by the isometric deformable convolution block branch is .

[0050] Step 2: Concatenate the perceptual features of the reference image block and the distorted image block to form a reference image block sequence and a distorted image block sequence. Interactively measure the similarity between the reference image block sequence and the distorted image block sequence through distortion perception to obtain a similarity feature:

[0051] In this step, the features extracted by the dual-branch feature extractor consisting of Swin Transformer and isometric deformable convolution block are fused for each reference image block and distorted image block. Since the sizes of the reference perception features and distortion perception features extracted by the Swin Transformer branch are inconsistent with those extracted by the isometric deformable convolution block branch, it is necessary to fuse the features of size of and Upsampled to and size is of and The specific relationship is as follows:

[0052] ;

[0053] in, represents the upsampling operation, represents the vector concatenation operation, Indicates The perceptual features of the reference image blocks after dual-branch fusion of Swin Transformer and isometric deformable convolutional blocks, Indicates The perceptual features of the distorted image blocks after dual-branch fusion of Swin Transformer and isometric deformable convolutional blocks. and The feature size after dual-branch fusion is .

[0054] In this step, the perceptual features of the reference image block and the distorted image block are respectively concatenated to form a reference image block sequence and a distorted image block sequence, and the specific relationship is as follows:

[0055] ;

[0056] in, It represents the perceptual features of the first reference image block after the dual-branch fusion of Swin Transformer and isometric deformable convolutional block. It represents the perceptual features of the first distorted image block after the dual-branch fusion of Swin Transformer and isometric deformable convolution block. Indicates The perceptual features of the reference image blocks after dual-branch fusion of Swin Transformer and isometric deformable convolutional blocks, Indicates The perceptual features of the distorted image blocks after dual-branch fusion of SwinTransformer and isometric deformable convolution blocks, represents the reference image block sequence, Represents the distorted image block sequence. The 10 reference image block features and the distorted image block features are concatenated to form and The size is .

[0057] In this step, the similarity between the reference image block sequence and the distorted image block sequence is measured through distortion-aware interaction to obtain similarity features. The specific steps are as follows:

[0058] The reference image block sequence and the distorted image block sequence are sent to the distortion perception interaction module. In order to reduce the amount of calculation, a The convolution of the size is applied to the reference image patch sequence and distorted image block sequence Dimensionality reduction is performed, and the sequence feature size is , the reduced features are passed through the multi-head attention module respectively, the features output by the multi-head attention module are sent to the cross attention module together with the difference between the reference image block sequence and the distorted image block sequence, and the features output by the cross attention module are added to the features output by the multi-head attention module to obtain the features finally generated by the multi-head attention and cross attention modules. Specifically, there is the following relationship:

[0059] ;

[0060] in, represents the multi-head attention module, represents the criss-cross attention module, represents the reference features generated by the reference image block sequence after the multi-head attention and cross-attention modules, Represents the distorted features generated by the distorted image block sequence after the multi-head attention and cross-attention modules. Reference features and distortion characteristics The size is .

[0061] The reference feature and the distortion feature are fused to obtain the similarity feature, which has the following relationship:

[0062] ;

[0063] in, represents the average pooling operation, represents the element-wise multiplication operation, Represents the similarity feature, with a size of .

[0064] Step 3: Perform block position attention calculation on the similarity features to achieve information interaction of similarity features in the horizontal and vertical directions, enhance the global information expression ability in the feature map and the sensitivity of the model to spatial position information, and obtain the similarity features after information interaction;

[0065] In this step, in order to reduce the amount of calculation, the similarity feature is first Use one Large and small convolutions for similarity features Perform dimensionality reduction and similarity features after dimensionality reduction The size is .

[0066] In this step, block position attention calculation is performed on the similarity features to achieve information interaction of similarity features in the horizontal and vertical directions, enhance the global information expression ability in the feature map and the sensitivity of the model to spatial position information, and obtain the similarity features after information interaction. The specific steps include:

[0067] Linear projection is performed on similarity features to generate query sequence, key sequence, and value sequence, which are specifically expressed as follows:

[0068] ;

[0069] ;

[0070] ;

[0071] in, represents different linear layer weight matrices, Represent query sequence, key sequence, and value sequence respectively;

[0072] First, add position encoding to the key sequence, and then expand the query sequence and the key sequence in the vertical direction and the horizontal direction to obtain the weight matrix in the vertical direction and the weight matrix in the horizontal direction, which are specifically expressed as follows:

[0073] ;

[0074] ;

[0075] in, Indicates the expansion of the query sequence in the vertical direction, represents the vertical expansion and transposition of the key sequence, Indicates the horizontal expansion of the query sequence. represents the horizontal expansion and transposition of the key sequence, represents the weight matrix in the vertical direction, Represents the weight matrix in the horizontal direction;

[0076] The weight matrix in the vertical direction and the weight matrix in the horizontal direction are concatenated to obtain the final weight matrix, which is specifically expressed as follows:

[0077] ;

[0078] in, represents the normalization function, Represents the final weight matrix.

[0079] The value sequence is multiplied by the final weight matrix in the corresponding direction in the horizontal and vertical directions, and then the weighted sum is performed to generate an enhanced feature map. The enhanced feature map is added to the similarity feature through the residual connection to form the final output, and the similarity feature after information interaction is obtained, which is specifically expressed as follows:

[0080] ;

[0081] in, represents the expansion and transposition of the weight matrix in the vertical direction, represents the horizontal expansion and transposition of the weight matrix, Representing similarity features The vertical expansion of the value sequence of Representing similarity features The horizontal expansion of the value sequence of represents the weight parameter, Represents the similarity features after information interaction.

[0082] Step 4: Perform a regression operation on the similarity features after information interaction to obtain the perceptual quality score of the non-uniformly distorted panoramic image.

[0083] In this step, a regression operation is performed on the similarity features after information interaction to obtain the perceptual quality score of the non-uniformly distorted panoramic image. Specifically, the following relationship exists:

[0084] ;

[0085] in, represents the flattening operation, represents a linear layer, Represents the perceptual quality score of the panoramic image.

[0086] The above method is implemented based on a full-reference panoramic image quality assessment model. In executing the above steps 1 to 4, the training method of the full-reference panoramic image quality assessment model includes the following training steps:

[0087] Acquire training data, the training data including several image blocks of the panoramic image and the distorted panoramic image. In this embodiment, 10 image blocks are used. Steps 1 to 4 are repeated with the training data as input to obtain a predicted quality score.

[0088] The mean square error is used as the loss function for quality score prediction, and the subjective score and predicted quality score are used to construct The loss function has the following relationship:

[0089] ;

[0090] in, and They are the first Predicted quality scores and subjective scores for panoramic images;

[0091] Input the loss function into the Adam optimizer for optimization, set the Adam optimizer weight decay strategy and learning parameters, where the learning rate is set to ; Weight decay strategy, the decay rate is ;

[0092] The performance of the full-reference panoramic image quality assessment model is improved by updating weights and learning parameters to minimize the loss.

[0093] Please refer to Figure 3 This embodiment further provides a full-reference panoramic image quality assessment system based on block sequence similarity, wherein the system applies the panoramic image quality assessment method based on block sequence similarity as described above, and the system includes:

[0094] Image block sampling module, used to:

[0095] Obtaining a reference image block and a distorted image block rich in semantic information from the reference panoramic image and the distorted panoramic image;

[0096] A dual-branch feature extractor is constructed based on Swin Transformer and isometric deformable convolution blocks. The dual-branch feature extractor is used to extract perceptual features of the reference image block and the distorted image block respectively to obtain the perceptual features of the reference image block and the distorted image block.

[0097] Distortion-aware interaction module for:

[0098] The perceptual features of the reference image block and the distorted image block are concatenated to form a reference image block sequence and a distorted image block sequence, and the similarity between the reference image block sequence and the distorted image block sequence is measured through distortion perception interaction to obtain a similarity feature;

[0099] Position attention module, used to:

[0100] Perform block position attention calculation on similarity features to achieve information interaction of similarity features in the horizontal and vertical directions, enhance the global information expression ability in the feature map and the model's sensitivity to spatial position information, and obtain similarity features after information interaction;

[0101] Quality regression module for:

[0102] The similarity features after information interaction are regressed to obtain the perceptual quality score of the non-uniformly distorted panoramic image.

[0103] The present invention can also evaluate the model performance in the following ways:

[0104] First, the average subjective score (MOS) of all data in the non-uniform distortion panoramic image database is obtained as the label data. The label data is represented by the following relationship:

[0105] ;

[0106] in, Indicates The experience quality scores given by the subjects for the non-uniformly distorted panoramic images, Indicates the number of people participating in the experiment to evaluate the quality of non-uniformly distorted panoramic images.

[0107] The perceptual quality score of the obtained non-uniformly distorted panoramic image is compared with The scores are compared and calculated to obtain the various indicators of the model. The test indicators include the following four types:

[0108] Forecast monotonicity indicators, including the Spearman correlation coefficient (SRCC), are specifically expressed as:

[0109] ;

[0110] in, represents the number of distorted images in the data, Indicates The difference between the subjective score of an image and the objective predicted score.

[0111] Forecast accuracy indicators, including the Pearson correlation coefficient (PLCC), are specifically expressed as:

[0112] ;

[0113] in, and Respectively represent The subjective score and objective prediction score of each image, and are the average of subjective scores and the average of objective prediction scores, respectively.

[0114] The prediction error degree index, including the root mean square error (RMSE), is specifically expressed as:

[0115] ;

[0116] in, and Respectively represent The subjective score and objective prediction score of each image, Indicates the number of distorted images in the data.

[0117] In summary, the present invention has conducted extensive experiments in the non-uniform distortion panoramic image quality evaluation database. The results show that this method has significant advantages in evaluating the quality of panoramic images. Each module plays a good role and has good robustness to non-uniform distortion.

[0118] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0119] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0120] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A full-reference panoramic image quality assessment method based on block sequence similarity, characterized in that: The method comprises the following steps: Step 1, obtaining a reference image block and a distorted image block rich in semantic information from the reference panoramic image and the distorted panoramic image; A dual-branch feature extractor is constructed based on Swin Transformer and isometric deformable convolution blocks. The dual-branch feature extractor is used to extract perceptual features of the reference image block and the distorted image block respectively, so as to obtain the perceptual features of the reference image block and the distorted image block. The dual-branch feature extractor is used to extract the perceptual features of the reference image block and the distorted image block respectively, so as to obtain the perceptual features of the reference image block and the distorted image block, which specifically includes the following steps: The Swin Transformer network and the first equidistant deformable convolution are used to construct the Swin Transformer branch, and the perceptual features of the reference image block and the distorted image block are extracted through the Swin Transformer branch respectively; The isometric deformable convolution block branch is constructed by using the second isometric deformable convolution, and the perceptual features extracted from the reference image block and the distorted image block are respectively extracted through the isometric deformable convolution block branch; Step 2: splicing the perceptual features of the reference image block and the distorted image block to form a reference image block sequence and a distorted image block sequence, and measuring the similarity between the reference image block sequence and the distorted image block sequence through distortion perception interaction to obtain a similarity feature; Step 3: Perform block position attention calculation on the similarity features to achieve information interaction of similarity features in the horizontal and vertical directions, enhance the global information expression ability in the feature map and the sensitivity of the model to spatial position information, and obtain the similarity features after information interaction; Step 4: Perform a regression operation on the similarity features after information interaction to obtain a perceptual quality score of the non-uniformly distorted panoramic image; In step 2, the perceptual features of the reference image block and the distorted image block are spliced ​​to form a reference image block sequence and a distorted image block sequence, which specifically includes the following steps: The features extracted by the dual-branch feature extractor composed of SwinTransformer and isometric deformable convolution blocks for each reference image block and distorted image block are fused. Specifically, the following relationship exists: ； in, represents the upsampling operation, represents the vector concatenation operation, Indicates The perceptual features of the reference image blocks after dual-branch fusion of Swin Transformer and isometric deformable convolutional blocks, Indicates The perceptual features of the distorted image blocks after dual-branch fusion of Swin Transformer and isometric deformable convolutional blocks; The perceptual features of the reference image block and the distorted image block are respectively concatenated to form a reference image block sequence and a distorted image block sequence, and the specific relationship is as follows: ； in, It represents the perceptual features of the first reference image block after the dual-branch fusion of Swin Transformer and isometric deformable convolutional block. It represents the perceptual features of the first distorted image block after the dual-branch fusion of Swin Transformer and isometric deformable convolution block. Indicates The perceptual features of the reference image blocks after dual-branch fusion of Swin Transformer and isometric deformable convolutional blocks, Indicates The perceptual features of the distorted image blocks after dual-branch fusion of SwinTransformer and isometric deformable convolution blocks, represents the reference image block sequence, represents a sequence of distorted image blocks; The similarity between the reference image block sequence and the distorted image block sequence is measured interactively through distortion perception to obtain similarity features, which specifically includes the following steps: The reference image block sequence and the distorted image block sequence are respectively reduced in dimension by convolution operation, and the reduced features are respectively passed through the multi-head attention module. The features output by the multi-head attention module are respectively sent to the cross attention module together with the difference between the reference image block sequence and the distorted image block sequence. The features output by the cross attention module are added to the features output by the multi-head attention module to obtain the features finally generated by the multi-head attention and cross attention modules. Specifically, there is the following relationship: ； in, represents the convolution operation, represents the multi-head attention module, represents the criss-cross attention module, represents the reference features generated by the reference image block sequence after the multi-head attention and cross-attention modules, Represents the distorted features generated by the distorted image block sequence after the multi-head attention and cross-attention modules; The reference feature and the distortion feature are fused to obtain the similarity feature, which has the following relationship: ； in, represents the average pooling operation, express Activation function, represents the element-wise multiplication operation, Represents similarity features.

2. The full-reference panoramic image quality assessment method based on block sequence similarity according to claim 1, characterized in that: In step 1, a Swin Transformer branch is constructed using a Swin Transformer network and a first equidistant deformable convolution, and the perceptual features of the reference image block and the distorted image block are respectively extracted through the Swin Transformer branch. Specifically, the following relationship exists: ； in, Indicates reference image blocks, Indicates Distorted image blocks, represents the parameters of the SwinTransformer pre-trained network, represents the Swin Transformer network, represents an isometric deformable convolution operation, represents the number of image blocks, Indicates The perceptual features extracted by the Swin Transformer branch of the reference image blocks, Indicates The perceptual features of the distorted image blocks extracted by the Swin Transformer branch; The second equidistant deformable convolution is used to construct an equidistant deformable convolution block branch, and the perceptual features extracted from the reference image block and the distorted image block are extracted through the equidistant deformable convolution block branch. Specifically, the following relationship exists: ； in, represents the isometric deformable convolutional block branch, Indicates The perceptual features extracted by the equidistant deformable convolutional block branches of the reference image blocks are: Indicates The perceptual features of the distorted image patches are extracted by the equidistant deformable convolutional block branch.

3. The full-reference panoramic image quality assessment method based on block sequence similarity according to claim 1, characterized in that: In step 3, block position attention calculation is performed on the similarity features to achieve information interaction of the similarity features in the horizontal and vertical directions, enhance the global information expression ability in the feature map and the sensitivity of the model to spatial position information, and obtain the similarity features after information interaction, which specifically includes the following steps: Linear projection is performed on similarity features to generate query sequence, key sequence, and value sequence, which are specifically expressed as follows: ； ； ； in, represents different linear layer weight matrices, Represent query sequence, key sequence, and value sequence respectively; First, add position encoding to the key sequence, and then expand the query sequence and the key sequence in the vertical direction and the horizontal direction to obtain the weight matrix in the vertical direction and the weight matrix in the horizontal direction, which are specifically expressed as follows: ； ； in, Indicates the expansion of the query sequence in the vertical direction, represents the vertical expansion and transposition of the key sequence, Indicates the horizontal expansion of the query sequence. represents the horizontal expansion and transposition of the key sequence. represents the weight matrix in the vertical direction, Represents the weight matrix in the horizontal direction; The weight matrix in the vertical direction and the weight matrix in the horizontal direction are concatenated to obtain the final weight matrix, which is specifically expressed as follows: ； in, represents the normalization function, represents the final weight matrix; The value sequence is multiplied by the final weight matrix in the corresponding direction in the horizontal and vertical directions, and then the weighted sum is performed to generate an enhanced feature map. The enhanced feature map is added to the similarity feature through the residual connection to form the final output, and the similarity feature after information interaction is obtained, which is specifically expressed as follows: ； in, represents the expansion and transposition of the weight matrix in the vertical direction, represents the horizontal expansion and transposition of the weight matrix, Representing similarity features The vertical expansion of the value sequence of Representing similarity features The horizontal expansion of the value sequence of represents the weight parameter, Represents the similarity features after information interaction.

4. The full-reference panoramic image quality assessment method based on block sequence similarity according to claim 3, characterized in that: In step 4, a regression operation is performed on the similarity features after information interaction to obtain the perceptual quality score of the non-uniformly distorted panoramic image, which specifically exists in the following relationship: ； in, represents the flattening operation, represents a linear layer, Represents the perceptual quality score of the panoramic image.

5. The full-reference panoramic image quality assessment method based on block sequence similarity according to claim 4, characterized in that: The method is implemented based on a full-reference panoramic image quality assessment model. In executing the above steps 1 to 4, the training method of the full-reference panoramic image quality assessment model includes the following training steps: Acquire training data, the training data including a plurality of image blocks of the panoramic image and the distorted panoramic image, and repeat steps 1 to 4 with the training data as input to obtain a predicted quality score; The mean square error is used as the loss function for quality score prediction, and the subjective score and predicted quality score are used to construct The loss function has the following relationship: ； in, and They are the first Predicted quality scores and subjective scores for panoramic images; The loss function is input into the Adam optimizer for optimization, and the weight decay strategy and learning parameters of the Adam optimizer are set; the loss is minimized by updating the weights and learning parameters to improve the performance of the full-reference panoramic image quality evaluation model.

6. A full-reference panoramic image quality assessment system based on block sequence similarity, characterized in that: The system applies the full-reference panoramic image quality assessment method based on block sequence similarity according to any one of claims 1 to 5, and the system comprises: Image block sampling module, used to: Obtaining a reference image block and a distorted image block rich in semantic information from the reference panoramic image and the distorted panoramic image; A dual-branch feature extractor is constructed based on Swin Transformer and isometric deformable convolution blocks. The dual-branch feature extractor is used to extract perceptual features of the reference image block and the distorted image block respectively, so as to obtain the perceptual features of the reference image block and the distorted image block. Distortion-aware interaction module for: The perceptual features of the reference image block and the distorted image block are concatenated to form a reference image block sequence and a distorted image block sequence, and the similarity between the reference image block sequence and the distorted image block sequence is measured through distortion perception interaction to obtain a similarity feature; Position attention module, used to: Perform block position attention calculation on similarity features to achieve information interaction of similarity features in the horizontal and vertical directions, enhance the global information expression ability in the feature map and the model's sensitivity to spatial position information, and obtain similarity features after information interaction; Quality regression module for: The similarity features after information interaction are regressed to obtain the perceptual quality score of the non-uniformly distorted panoramic image.

Images