Dynamic cross-level distortion information fusion no-reference panoramic image quality assessment method

Abstract

The present invention proposes a method for evaluating the quality of a dynamic cross-level distortion information fused reference-free panoramic image, which includes: inputting a panoramic image into a target tracking model, performing an adaptive equatorial prior sampling operation on the panoramic image through an adaptive prior sampling module, and obtaining a set of image blocks; processing the set of image blocks through a backbone network module, and obtaining an output feature map of the image blocks containing image distortion information and visual features; assigning different contribution weights to the output feature map of each image block through an image block attention module, and obtaining a weighted feature vector; performing a quality regression operation on the weighted feature vector through a quality regression module, and obtaining quality perception scores of different image blocks and adding and averaging them, and further obtaining a quality perception score of an equirectangular projection format image. The present invention can solve the problem of irregular distortion inherent in equirectangular projection format panoramic images.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and multimedia digital image processing, and in particular to a dynamic cross-level distortion information fusion reference-free panoramic image quality evaluation method. Background Art

[0002] Virtual reality (VR) provides a simulated environment, which enables users to freely enjoy various contents in an immersive experience through head-mounted displays. Panoramic images are an important medium for virtual reality visual presentation. Their quality may be degraded during acquisition, transmission, processing, and storage. The degradation of panoramic image quality not only greatly reduces the end-user's Quality of Experience (QoE), but also affects the availability of panoramic images. Accurately evaluating the quality of panoramic images is crucial for algorithm optimization and system optimization.

[0003] At present, most of the panoramic image quality evaluation databases are panoramic images with uniform distortion. In the real world, panoramic images will inevitably produce uniform distortion in the process of compression, transmission and display, and will also introduce non-uniform distortion in the process of acquisition and stitching. This degradation of local areas is more obvious than the surrounding areas, and even seriously affects the overall immersive experience of users. When the panoramic image quality evaluation algorithm based on uniform distortion processes non-uniformly distorted panoramic images, its evaluation results do not conform to the user's real evaluation. Compared with the panoramic image quality evaluation based on uniform distortion, the panoramic image quality evaluation research based on real distortion has more important academic value and application prospects.

[0004] Although current models have made great progress in performance, they often use powerful feature extraction backbone networks or introduce sophisticated module designs, resulting in high model complexity. Combined with the current generally small capacity of panoramic image quality datasets, this undoubtedly increases the difficulty of model training and the risk of overfitting. Summary of the invention

[0005] In view of the above situation, the main purpose of the present invention is to propose a dynamic cross-level distortion information fusion no-reference panoramic image quality evaluation method to solve the above technical problems.

[0006] The present invention proposes a method for evaluating the quality of a dynamic cross-level distortion information fusion panoramic image without reference, the method comprising the following steps:

[0007] Step 1: Input the panoramic image into the target tracking model, and perform adaptive equatorial prior sampling operation on the panoramic image through the adaptive prior sampling module to obtain an image block set;

[0008] Step 2: Process the image block set through the backbone network module to obtain an output feature map of the block containing image distortion information and visual features;

[0009] Step 3: Assign different contribution weights to the output feature map of each block through the image block attention module, and obtain the weighted feature vector;

[0010] Step 4: Perform a quality regression operation on the weighted feature vector through the quality regression module to obtain the quality perception scores of different image blocks and add them up to take the average, and further obtain the quality perception score of the equirectangular projection format image.

[0011] Compared with the prior art, the present invention has the following beneficial effects:

[0012] 1. The image block sampling method based on the adaptive equatorial prior of the present invention can effectively represent the key content of the panoramic image and ensure more adequate coverage of the area rich in semantic information;

[0013] 2. The present invention uses SwinTransformer and multi-level joint distortion perception module, combined with deformable convolution to handle irregular geometric deformations in ERP images, and uses dynamic inter-layer interaction to achieve effective feature expression from different scales and layers;

[0014] 3. The present invention constructs an image block attention module to assign different contribution weights to each image block feature and adjust the degree of influence of different image blocks on the overall quality;

[0015] 4. Compared with other methods, the present invention has excellent performance over other methods under the condition of controlling model parameters, and has good portability, which is conducive to the effective application of this technology in industrial environments.

[0016] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of the dynamic cross-level distortion information fusion no-reference panoramic image quality assessment method proposed by the present invention;

[0018] Figure 2 Schematic diagram of the overall framework of the dynamic cross-level distortion information fusion no-reference panoramic image quality assessment method proposed in the present invention. DETAILED DESCRIPTION

[0019] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0020] These and other aspects of the embodiments of the present invention will be apparent with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0021] See also Figure 1 and Figure 2 The embodiment of the present invention proposes a method for evaluating the quality of a dynamic cross-level distortion information fusion panoramic image without reference, and the method comprises the following steps:

[0022] Step 1: Input the panoramic image into the target tracking model, and perform adaptive equatorial prior sampling operation on the panoramic image through the adaptive prior sampling module to obtain an image block set;

[0023] In step 1, the panoramic image is input into the target tracking model, and the adaptive prior sampling module performs an adaptive equatorial prior sampling operation on the panoramic image to obtain an image block set. The specific steps are as follows:

[0024] The coordinates after the given longitude and latitude conversion ,coordinate Obey the equatorial prior distribution, coordinates Follow uniform distribution;

[0025] Set the latitude division factor And the number of sampling grids , and the original sampling image is gridded. The number of grids in the low-latitude area of ​​the original sampling image is , the number of high-latitude grids of the original sampled image .

[0026] The parameters of the sampling block are determined by setting the longitude sampling factor and the latitude sampling factor. The corresponding relationship is:

[0027] ;

[0028] in, and Respectively represent the width and height of the sampled image block, and Respectively represent the width and height of the sampled panoramic image block, represents the sampling factor in the longitude direction, Indicates the sampling factor in the latitude direction;

[0029] Each grid block is randomly sampled in the longitude and latitude directions to obtain a set of image block coordinates containing visual information, and then the panoramic image is sliced ​​according to the parameters of the sampling block and the coordinates after the longitude and latitude conversion to obtain a set of image blocks;

[0030] Normalize the collection of image patches to improve the performance and stability of the model when processing and analyzing images.

[0031] Step 2: Process the image block set through the backbone network module to obtain an output feature map of the block containing image distortion information and visual features;

[0032] In step 2, the image block set is processed by the backbone network module to obtain an output feature map of the block containing image distortion information and visual features. The specific steps are as follows:

[0033] Based on the image block set, the output feature maps of different stages and the output feature maps of the window division stage are obtained through the Swin Transformer pre-trained network. The relationship between the corresponding processes is:

[0034] ;

[0035] in, Represents the output feature maps of Swin Transformer at different stages and the output feature maps of the window division stage, Indicates processing through the Swin Transformer pre-trained network, represents a set of image blocks, represents the parameters of the Swin Transformer pre-trained network, Indicates different stages of the backbone network;

[0036] The output feature maps of the Swin Transformer at different stages and the output feature maps at the window division stage are adjusted by deformable convolution to obtain the feature maps adjusted by deformable convolution. The relationship between the corresponding processes is:

[0037] ;

[0038] in, Indicates The feature map adjusted by deformable convolution in the first stage, represents a deformable convolution operation;

[0039] The feature map adjusted by the deformable convolution is added to the adjacent feature map in an element-by-element manner to obtain the output feature map adjusted by the deformable convolution. The corresponding process has the following relationship:

[0040] ;

[0041] in, Represents Swin Transformer No. The feature map of the first stage and the The feature map after element-wise addition of the feature maps of the stages, It represents a 2D convolution operation with a convolution kernel of 1×1 and a stride of 2. represents an element-wise addition operation, It represents the output feature map after the fused feature map is adjusted by deformable convolution;

[0042] The adjacent output feature maps adjusted by deformable convolution are element-wise added to obtain the output feature map of the multi-level joint distortion perception module. The corresponding process has the following relationship:

[0043] ;

[0044] in, represents the output feature map of the multi-level joint distortion perception module, represents the distortion-aware convolution operation, It represents the output feature map after the fusion of the feature map of the backbone network window division stage and the feature map of the first stage. It represents the output feature map after the first stage feature map and the second stage feature map are fused after deformable convolution processing. It represents the output feature map after the second stage feature map and the third stage feature map are fused after deformable convolution processing;

[0045] The output feature map of the multi-stage joint distortion perception module is added to the output feature map of the fourth stage joint distortion perception module by element-by-element addition to obtain the output feature map of the block. The relationship between the corresponding processes is:

[0046] ;

[0047] in, Indicates The output feature map of the image patch is Represents the output feature map of the joint distortion perception module in the fourth stage, Indicates the number of image blocks.

[0048] Step 3: Assign different contribution weights to the output feature map of each block through the image block attention module, and obtain the weighted feature vector;

[0049] In step 3, different contribution weights are assigned to the output feature map of each block through the image block attention module, and the weighted feature vector is obtained. The specific steps are as follows:

[0050] The output feature map of the block is averaged and pooled to obtain the feature vector of the image block. The corresponding process has the following relationship:

[0051] ;

[0052] in, No. The feature vector of the image block, represents the average pooling operation;

[0053] After the feature vectors of the image blocks are concatenated, they are multiplied by the weight matrix to obtain a weighted feature vector set. The corresponding process has the following relationship:

[0054] ;

[0055] in, represents the weighted feature vector set, and ; represents matrix multiplication, represents the weight matrix, represents the vector concatenation operation, Represents element-wise multiplication.

[0056] Step 4: Perform a quality regression operation on the weighted feature vector through a quality regression module to obtain the quality perception scores of different image blocks and add and average them, and further obtain the quality perception score of the equirectangular projection format image;

[0057] In step 4, the quality regression module performs a quality regression operation on the weighted feature vector to obtain the quality perception scores of different image blocks and add and average them, and further obtain the quality perception score of the equirectangular projection format image. The corresponding relationship in the process is:

[0058] ;

[0059] in, Represents the perceived quality score of an image in equirectangular projection format, It means processed by the fully connected layer. represents the fully connected layer parameters, Indicates A weighted feature vector.

[0060] In executing the above steps 1 to 4, the corresponding training method includes the following training steps:

[0061] Based on the nested norm loss, a loss function is constructed using a set of predicted quality scores and a given set of subjective quality scores. The construction process of the loss function has the following relationship:

[0062] ;

[0063] in, represents the nested norm loss, represents the regularization factor, represents the hyperparameter, and denote the predicted quality score and the subjective quality score respectively, , ; represents the number of mass scores within a group, represents the mean function, and denote the standard deviation of the predicted quality score and the subjective quality score, respectively;

[0064] Compare the prediction results with the MOS score to calculate the various indicators of the model. The relationship between the corresponding process is:

[0065] ;

[0066] in, Indicates The mean opinion score of the images, Indicates Picture courtesy of The quality experience score given by the annotators, represents the total number of annotators;

[0067] Three IQA model evaluation indicators are used, namely Pearson linear correlation coefficient (PLCC), Spearman correlation coefficient (SRCC) and root mean square error (RMSE). Among them, PLCC and RMSE are used to evaluate the accuracy of the prediction, while SRCC is used to evaluate the consistency of the prediction monotonicity. Higher PLCC and SRCC values ​​and lower RMSE values ​​indicate higher prediction accuracy and consistency. In addition to SRCC, PLCC and RMSE are calculated according to the five-parameter nonlinear mapping. The specific mapping functions are as follows:

[0068] ;

[0069] in, and denote the quality score of the prediction and the objective score of the mapping, respectively. , , , and are fitting parameters.

[0070] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0071] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0072] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A dynamic cross-level distortion information fusion no-reference panoramic image quality assessment method, characterized in that: The method is implemented based on a target tracking model, which is composed of an adaptive prior sampling module, a backbone network module, an image block attention module and a quality regression module. The method includes the following steps: Step 1: Input the panoramic image into the target tracking model, and perform adaptive equatorial prior sampling operation on the panoramic image through the adaptive prior sampling module to obtain an image block set; Step 2: Process the image block set through the backbone network module to obtain an output feature map of the block containing image distortion information and visual features; Step 3: Assign different contribution weights to the output feature map of each block through the image block attention module, and obtain the weighted feature vector; Step 4: Perform a quality regression operation on the weighted feature vector through a quality regression module to obtain the quality perception scores of different image blocks and add and average them, and further obtain the quality perception score of the equirectangular projection format image; The panoramic image is input into the target tracking model, and an adaptive equatorial prior sampling operation is performed on the panoramic image through the adaptive prior sampling module to obtain an image block set. The specific steps are as follows: Given a coordinate after longitude and latitude conversion, the coordinate after longitude and latitude conversion obeys an equatorial prior distribution in the latitude direction and obeys a uniform distribution in the longitude direction; Set the latitude division factor and the number of sampling grids, and perform grid division on the original sampling image; The parameters of the sampling block are determined by setting the longitude sampling factor and the latitude sampling factor. The corresponding relationship is: ； in, and Respectively represent the width and height of the sampled image block, and Respectively represent the width and height of the sampled panoramic image block, represents the sampling factor in the longitude direction, Indicates the sampling factor in the latitude direction; Each grid block is randomly sampled in the longitude and latitude directions to obtain a set of image block coordinates containing visual information, and then the panoramic image is sliced ​​according to the parameters of the sampling block and the coordinates after the longitude and latitude conversion to obtain a set of image blocks; Normalize the set of image patches to improve the performance and stability of the model when processing and analyzing images; In step 2, the image block set is processed by the backbone network module to obtain an output feature map of the image block containing image distortion information and visual features. The specific steps are as follows: Based on the image block set, the output feature maps of different stages and the output feature maps of the window division stage are obtained through the Swin Transformer pre-trained network. The relationship between the corresponding processes is: ； in, Represents the output feature maps of Swin Transformer at different stages and the output feature maps of the window division stage, Indicates processing through the Swin Transformer pre-trained network, represents a set of image blocks, represents the parameters of the SwinTransformer pre-trained network, Indicates different stages of the backbone network; The output feature maps of the Swin Transformer at different stages and the output feature maps at the window division stage are adjusted by deformable convolution to obtain the feature maps adjusted by deformable convolution. The relationship between the corresponding processes is: ； in, Indicates The feature map adjusted by deformable convolution in the first stage, represents a deformable convolution operation; In the step of processing the image block set by the backbone network module to obtain an output feature map of the image block containing image distortion information and visual features, the method further includes: The feature map adjusted by the deformable convolution is added to the adjacent feature map in an element-by-element manner to obtain the output feature map adjusted by the deformable convolution. The corresponding process has the following relationship: ； in, Represents Swin Transformer No. The feature map of the first stage and the The feature map after element-wise addition of the feature maps of the stages, Indicates a 2D convolution operation with a convolution kernel of 1×1 and a stride of 2; represents an element-wise addition operation, It represents the output feature map after the fused feature map is adjusted by deformable convolution; The adjacent output feature maps adjusted by deformable convolution are element-wise added to obtain the output feature map of the multi-level joint distortion perception module. The corresponding process has the following relationship: ； in, represents the output feature map of the multi-level joint distortion perception module, Indicates a distortion-aware convolution operation. It represents the output feature map after the fusion of the feature map of the backbone network window division stage and the feature map of the first stage. It represents the output feature map after the first stage feature map and the second stage feature map are fused after deformable convolution processing. It represents the output feature map after the second stage feature map and the third stage feature map are fused after deformable convolution processing; The output feature map of the multi-stage joint distortion perception module is added element by element to the output feature map of the joint distortion perception module of the fourth stage to obtain the output feature map of the block.

2. The dynamic cross-level distortion information fusion no-reference panoramic image quality assessment method according to claim 1, characterized in that: In the step of processing the image block set by the backbone network module to obtain an output feature map of the image block containing image distortion information and visual features, the method further includes: The output feature map of the multi-stage joint distortion perception module is added to the output feature map of the fourth stage joint distortion perception module by element-by-element addition to obtain the output feature map of the block. The relationship between the corresponding processes is: ； in, Indicates The output feature map of the image patch is Represents the output feature map of the joint distortion perception module in the fourth stage, Indicates the number of image blocks.

3. The dynamic cross-level distortion information fusion no-reference panoramic image quality assessment method according to claim 2, characterized in that: In step 3, different contribution weights are assigned to the output feature map of each block through the image block attention module, and a weighted feature vector is obtained. The specific steps are as follows: The output feature map of the block is averaged and pooled to obtain the feature vector of the image block. The corresponding process has the following relationship: ； in, No. The feature vector of the image block, represents the average pooling operation; After the feature vectors of the image blocks are concatenated, they are multiplied by the weight matrix to obtain a weighted feature vector set. The corresponding process has the following relationship: ； in, represents the weighted feature vector set, and ; represents matrix multiplication, represents the weight matrix, represents the vector concatenation operation, Represents element-wise multiplication.

4. The dynamic cross-level distortion information fusion no-reference panoramic image quality assessment method according to claim 3, characterized in that: In step 4, the quality regression module performs a quality regression operation on the weighted feature vector to obtain the quality perception scores of different image blocks and add and average them, and further obtain the quality perception score of the image in the equirectangular projection format. The relationship between the corresponding process is: ； in, Represents the perceived quality score of an image in equirectangular projection format, It means processed by the fully connected layer. represents the fully connected layer parameters, Indicates A weighted feature vector.

5. The dynamic cross-level distortion information fusion no-reference panoramic image quality assessment method according to claim 4, characterized in that: In executing the above steps 1 to 4, the corresponding training method includes the following training steps: Based on the nested norm loss, a loss function is constructed using a set of predicted quality scores and a given set of subjective quality scores. The construction process of the loss function has the following relationship: ； in, represents the nested norm loss, represents the regularization factor, represents the hyperparameter, and denote the predicted quality score and the subjective quality score respectively, and , ; represents the number of mass scores within a group, represents the mean function, and denote the standard deviation of the predicted quality score and the subjective quality score, respectively.

Images