A method for evaluating the quality of panoramic stitched images
By using SIFT-Flow and multivariate linear regression models in panoramic stitching images, the features of stitching viewports and significant viewports were extracted, and the problem of incomplete evaluation of distortion of panoramic stitching images in the prior art is solved, and more accurate images are achieved Quality evaluation.
Patent Information
- Application Number
- CN202210324929.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-30
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-03-30
AI Technical Summary
The existing image quality evaluation methods fail to effectively consider the impact of local errors such as misalignment, ghosting, deformation and other local errors on image quality in panoramic stitching images, and lack a comprehensive evaluation of distortion of stitching images, resulting in the objective evaluation results that are inconsistent with human eye perception. .
The panoramic stitching image quality evaluation method is used to establish the matching relationship between the stitching viewport and the significant viewport through the SIFT-Flow method, and the edge perception, deformation perception, color perception and information perception features are extracted, and the multivariate linear regression model is used for training and prediction. Evaluation of objective quality of images.
The correlation between objective evaluation results and subjective perception of human eyes is improved, and the distortion of panoramic stitching images is fully reflected, providing a more accurate quality evaluation.
Smart Images

Figure CN114972169B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an image quality evaluation technology, and in particular to a panoramic stitching image quality evaluation method. Background Art
[0002] Image stitching technology can combine multiple single-viewpoint images with overlapping parts into a wide-viewing angle and high-resolution panoramic image, which is a hot issue in the field of computer vision. Early image stitching technology used a standard lens to shoot single-viewpoint images with a viewing angle of about 50°. It required multiple single-viewpoint images in different directions to generate a full-range panoramic image, which not only put forward higher requirements on the shooting process, but also multiple image stitching operations are more likely to introduce image distortion. In recent years, fisheye cameras have attracted widespread attention due to their advantage of large viewing angles. Using fisheye cameras to shoot two fisheye images in opposite directions can obtain full-range scene information. However, the limited overlapping area between fisheye images puts higher requirements on geometric alignment, and inconsistent image capture settings will cause color distortion near the stitching seam. Therefore, designing an effective panoramic stitching image quality evaluation method can assist in the optimization of the stitching algorithm, which is of great significance to the development of virtual reality technology.
[0003] Although image quality evaluation methods are becoming more and more mature, existing image quality evaluation methods mainly consider the impact of diffusion distortion on image quality, while the distortion of stitched images mainly comes from local errors such as misalignment, ghosting, and deformation, which are not considered by traditional image quality evaluation methods. In addition, existing stitched image quality evaluation methods only consider the impact of some distortion attributes on image quality, but lack comprehensive consideration of stitched image distortion. Therefore, how to effectively extract visual features in the evaluation process so that the objective evaluation results are more consistent with the perception of the human visual system is a problem that needs to be solved in the objective quality evaluation of panoramic stitched images. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a method for evaluating the quality of panoramic stitching images, which can effectively improve the correlation between objective evaluation results and subjective perception of human eyes.
[0005] The technical solution adopted by the present invention to solve the above technical problem is: a method for evaluating the quality of panoramic stitching images, characterized by comprising the following steps:
[0006] Step 1: Select two panoramic stitching images obtained by stitching two fisheye images at different orientations, and use the panoramic stitching image obtained by stitching two fisheye images at 90° orientation and 270° orientation as the distorted image, and record it as I S , will I S The pixel value of the pixel point with coordinate position (u, v) is recorded as IS (u, v); The panoramic stitching image obtained by stitching the two fisheye images at 0° and 180° is used as the reference image and is recorded as I G , will I G The pixel value of the pixel point with coordinate position (u, v) is recorded as I G (u,v); where 1≤u≤W, 1≤v≤H, W represents I S and I G Width, H represents I S and I G Height;
[0007] Step 2: I S Perform projection transformation to obtain I S The corresponding N1 distorted spliced viewport images and N2 distorted significant viewport images are I S The corresponding i-th distorted spliced viewport image is recorded as Will I S The corresponding j-th distorted viewport image is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as
[0008] Similarly, for I G Perform projection transformation to obtain I G The corresponding N1 reference stitching viewport images and N2 reference salient viewport images are I G The corresponding i-th reference stitching viewport image is recorded as Will I G The corresponding j-th reference salient viewport image is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as
[0009] Among them, N1≥1, N2≥1, 1≤i≤N1, 1≤j≤N2;
[0010] Step 3: Use SIFT-Flow method to establish I G Each corresponding reference stitching viewport image and I S The matching relationship between each corresponding distorted spliced viewport image is obtained. GThe corresponding SIFT-Flow vector of each pixel in each reference stitched viewport image is and Using SIFT-Flow method to build and The matching relationship between The SIFT-Flow vector of each pixel in The mid-coordinate position is The SIFT-Flow vector of the pixel point is recorded as
[0011] Similarly, SIFT-Flow method is used to establish I G Each corresponding reference salient viewport image is S The matching relationship between each distorted viewport image is obtained by G The corresponding SIFT-Flow vector of each pixel in each reference salient viewport image is and Using SIFT-Flow method to build and The matching relationship between The SIFT-Flow vector of each pixel in The mid-coordinate position is The SIFT-Flow vector of the pixel point is recorded as
[0012] in, Used to indicate the horizontal direction. Used to indicate the vertical direction. express The horizontal offset of express The vertical offset of express The horizontal offset of express The vertical offset of
[0013] Step 4: Use simple linear iterative clustering method to G Each corresponding reference spliced viewport image is divided into superpixels. right Perform superpixel block division to obtain N seed non-overlapping superpixel blocks, The set of coordinate positions of all pixels in the ξth superpixel block is recorded as Then according to I GThe corresponding SIFT-Flow vector of each pixel in each reference stitching viewport image is determined by I G Each superpixel block in each corresponding reference stitching viewport image is in I S The corresponding superpixel blocks in each distorted spliced viewport image are according to The SIFT-Flow vector of each pixel in is determined in Zhongyu Each superpixel block in the matching superpixel block will be Zhongyu The set of coordinate positions of all pixels in the superpixel block matched by the ξth superpixel block in is recorded as Among them, N seed ≥1, 1≤ξ≤N seed ;
[0014] Similarly, a simple linear iterative clustering method is used to G Each corresponding reference salient viewport image is divided into superpixels. right Perform superpixel block division to obtain N seed non-overlapping superpixel blocks, The set of coordinate positions of all pixels in the ξth superpixel block is recorded as Then according to I G The corresponding SIFT-Flow vector of each pixel in each reference salient viewport image is determined by I G Each superpixel block in each corresponding reference salient viewport image is in I S The corresponding superpixel blocks in each distorted viewport image are according to The SIFT-Flow vector of each pixel in is determined in Zhongyu Each superpixel block in the matching superpixel block will be Zhongyu The set of coordinate positions of all pixels in the superpixel block matched by the ξth superpixel block in is recorded as
[0015] Step 5: According to I G Each superpixel block and I in the first principal component, the second principal component, and the third principal component of the local guided kernel feature tensor of the grayscale image of each reference stitching viewport image G Each superpixel block in the first principal component, the second principal component, and the third principal component of the local guided kernel feature tensor of the grayscale image of each corresponding reference spliced viewport image is in I SThe corresponding superpixel blocks in the first principal component, the second principal component, and the third principal component of the local guided kernel feature tensor of the grayscale image of each distorted spliced viewport image are calculated. S The edge-aware features of each superpixel block in each distorted spliced viewport image will be Zhongyu The edge-aware feature of the superpixel block matched by the ξth superpixel block in is recorded as in, The dimension of is 1×3, Indicated in The first principal component of the local guided kernel feature tensor of the grayscale image Zhongyu The first principal component of the local guided kernel feature tensor of the grayscale image The edge-aware features of the superpixel block matched by the ξ-th superpixel block in Indicated in The second principal component of the local guided kernel feature tensor of the grayscale image Zhongyu The second principal component of the local guided kernel feature tensor of the grayscale image The edge-aware features of the superpixel block matched by the ξ-th superpixel block in Indicated in The third principal component of the local guided kernel feature tensor of the grayscale image Zhongyu The third principal component of the local guided kernel feature tensor of the grayscale image The edge-aware features of the superpixel block matched by the ξ-th superpixel block in ;
[0016] Step 6: According to I G Each superpixel block and I in each corresponding reference spliced viewport image G Each superpixel block in each corresponding reference stitching viewport image is in I S The corresponding superpixel blocks in each distorted spliced viewport image are calculated S The deformation-aware features of each superpixel block in each distorted spliced viewport image will be Zhongyu The deformation-aware feature of the superpixel block matched by the ξth superpixel block in is recorded as in, The dimension of is 1×2, express The ξth superpixel block in Zhongyu The aspect ratio similarity of the superpixel block matched by the ξ-th superpixel block in , express The ξth superpixel block in Zhongyu The boundary similarity of the superpixel block matched by the ξ-th superpixel block in ;
[0017] Step 7: According to I G Each superpixel block and I in the H component, S component and V component of each corresponding reference spliced viewport image G Each superpixel block in the H component, S component, and V component of each corresponding reference stitching viewport image is in I S The corresponding superpixel blocks in the H component, S component, and V component of each distorted spliced viewport image are calculated. S The color perception features of each superpixel block in each distorted spliced viewport image will be Zhongyu The color perception feature of the superpixel block matched by the ξth superpixel block in is recorded as in, The dimension of is 1×3, Indicated in Zhongyu The color perception feature of the H channel of the superpixel block matched by the ξth superpixel block in Indicated in Zhongyu The color perception feature of the S channel of the superpixel block matched by the ξth superpixel block in express Zhongyu The color perception feature of the V channel of the superpixel block matched by the ξ-th superpixel block in ;
[0018] Step 8: I S The edge-aware features of all superpixel blocks in each corresponding distorted spliced viewport image are adaptively fused to obtain I S The corresponding edge-perceived features of each distorted spliced viewport image are The edge perception feature is recorded as Similarly, for I S The deformation perception features of all superpixel blocks in each distorted spliced viewport image are adaptively fused to obtain I S The corresponding deformation perception features of each distorted spliced viewport image are The deformation perception feature is recorded as to I S The color perception features of all superpixel blocks in each distorted spliced viewport image are adaptively fused to obtain I S The corresponding color perception features of each distorted spliced viewport image are The color perception characteristics of in, The dimension of is 1×3, The dimension of is 1×2, The dimension is 1×3;
[0019] Step 9: Pass I S Each corresponding distorted stitched viewport image and I G The mutual information value between each corresponding reference spliced viewport image is calculated S The corresponding information perception features of each distorted spliced viewport image are The information perception feature of in, express The information entropy of express The information entropy of express and The information entropy of the joint distribution, The dimension of is 1×1;
[0020] Step 10: According to I S The corresponding edge perception features, deformation perception features, color perception features, and information perception features of each distorted spliced viewport image are obtained. S The corresponding feature vector of each distorted spliced viewport image is The characteristic vector of in, Indicates that are concatenated to form a feature vector, The dimension is 1×9;
[0021] Step 11: Follow the process from step 5 to step 10 to obtain I in the same way. S The edge perception features, deformation perception features, and color perception features of each superpixel block in each distorted viewport image are obtained, thereby obtaining I S The corresponding edge perception features, deformation perception features, and color perception features of each distorted viewport image are obtained, and I S The corresponding information perception features of each distorted viewport image are finally obtained. S The corresponding feature vector of each distorted viewport image is The characteristic vector of in, express The edge perception features of The dimension of is 1×3, express The deformation perception characteristics of The dimension of is 1×2, express The color perception characteristics of The dimension of is 1×3, express Information perception characteristics, The dimension of is 1×1, The dimension is 1×9;
[0022] Step 12: According to I S The corresponding feature vectors and I of all distorted spliced viewport images S The corresponding feature vectors of all distorted viewport images are calculated. S The characteristic vector of is denoted as F, F = w T ×F T +w A ×F A ; Among them, F T represents the feature vector of the splicing area, F T The dimension of is 1×9, F A represents the salient region feature vector, F A The dimension of is 1×9, The dimension of F is 1×9, w T F T The weight, w A F A The weight, w T +w A =1,
[0023] Step 13: Select n' panoramic stitching images as distorted images and the cross-reference images corresponding to each panoramic stitching image as reference images to form an image set; then use the subjective quality evaluation method to obtain the average subjective score value of each distorted image in the image set, and record the average subjective score value of the i'th distorted image in the image set as MOS i' ; and according to the process of step 2 to step 12, the feature vector of each distorted image in the image set is obtained in the same way, and the feature vector of the i'th distorted image in the image set is recorded as F i' ; Then classify all distorted images in the image set according to the scene content, assuming that there are k' image classes; where n'>1, 1≤i'≤n', MOS i' ∈[0,7],F i' The dimension is 1×9, k'>1;
[0024] Step 14: Randomly select all distorted images of k'-1 image classes from the image set to form a training set; then the feature vectors and average subjective rating values of all distorted images in the training set form a training sample data set; then use multivariate linear regression as a machine learning method to train all feature vectors in the training sample data set, so that the error between the regression function value obtained after training and the average subjective rating value is minimized, and the optimal weight vector is fitted. and the optimal bias term Reuse the optimal weight vector and the optimal bias term Construct a multivariate linear regression training model for panoramic stitching images, denoted as The total number of distorted images in the training set is m, 1≤m<n', is a function representation, the superscript "T" indicates the transpose of a vector or matrix, F inp represents the input vector of the multivariate linear regression training model of the panoramic stitching image, is the feature vector of the distorted image, and F inp The dimension of is 1×9, The input vector F represents the multivariate linear regression training model of the panoramic stitching image. inp A linear function of
[0025] Step 15: All distorted images of the remaining 1 image class in the image set constitute a test set; then the feature vectors of all distorted images in the test set constitute a test sample data set; then, according to the constructed multivariate linear regression training model of the panoramic stitching image, each feature vector in the test sample data set is tested to predict the objective quality evaluation prediction value of the distorted image corresponding to each feature vector in the test sample data set, and the objective quality evaluation prediction value of the distorted image corresponding to the nth feature vector in the test sample data set is recorded as quality η , Among them, 1≤η≤n'-m, F η represents the nth feature vector in the test sample data set, F η The dimension of is 1×9, Indicates F η A linear function of
[0026] Step 16: Repeat steps 14 to 15 k' times in total, so that each distorted image in the image set belongs to the test set only once, and obtain the objective quality evaluation prediction value of each distorted image in the image set.
[0027] In the step 2, and and and The acquisition process is:
[0028] Step 2_1: I S Convert from the equirectangular coordinate system to the geometric coordinate system to obtain the corresponding spherical panoramic image, denoted as I sphere,S , will I sphere,S The pixel value of the pixel point with the coordinate position (φ,θ) is recorded as I sphere,S (φ,θ), Among them, φ is the longitude coordinate in the geometric coordinate system, θ is the latitude coordinate in the geometric coordinate system, -π≤φ≤π, φ=0 and φ=π correspond to the positions of the two seams;
[0029] Step 2_2: I sphere,S The coordinates of the spherical center points of the N1 stitching viewports are projected to obtain N1 distorted stitching viewport images. sphere,S The coordinates of the center point of the sphere along the i-th stitching viewport The i-th distorted spliced viewport image obtained by projection is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Among them, N1≥1, 1≤i≤N1, Represents the coordinate position Right now Represents the longitude coordinate of the spherical center point of the i-th stitching viewport, Represents the latitude coordinate of the spherical center point of the i-th stitching viewport, Represents the longitude coordinate of the i-th stitching viewport at the spherical sampling point, Represents the latitude coordinate of the i-th stitching viewport at the spherical sampling point, The step size is The step size is α i Represents the i-th distorted stitched viewport image The viewing angle size;
[0030] Step 2_3: Use panoramic convolution 360° single image saliency prediction method to extract I S The saliency map of A , will I A The pixel value of the pixel point with coordinate position (u, v) is I S The saliency value of the pixel with coordinate position (u, v) is recorded as I A (u,v); then for I AThe coordinate positions of all pixels in are arranged in descending order according to the pixel values of the pixels, and the set of arranged coordinate positions is recorded as {P1, P2, P3, ..., P W×H}; where P1 = (u1, v1) represents the first coordinate position after arrangement, i.e. I A The coordinate position of the pixel with the largest pixel value in the array, P2 = (u2, v2) represents the second coordinate position after arrangement, which is I A The coordinate position of the pixel with the second largest pixel value, P3 = (u3, v3) represents the third coordinate position after arrangement, which is I A The coordinate position of the pixel with the third largest pixel value, P W×H =(u W×H ,v W×H ) represents the W×Hth coordinate position after arrangement, which is I A The coordinate position of the pixel with the smallest pixel value;
[0031] Step 2_4: Select the coordinates of the spherical center point of the first significant viewport The specific process is:
[0032] Traverse {P1,P2,P3,...,P W×H}, transform P1 from the equirectangular coordinate system to the geometric coordinate system to obtain the transformed coordinate position, recorded as P1', P1'=(φ1,θ1), Then calculate the angular distance between P1' and the coordinates of the spherical center point of N1 stitching viewports, and replace P1' with the coordinates of the spherical center point of the i-th stitching viewport The angular distance is denoted as d i , Then get the minimum angle distance, recorded as d min , d min =min(d i |1≤i≤N1); then judge d min Is it greater than the threshold d? th , if d min Greater than the threshold d th , then P1' is taken as the coordinate of the spherical center point of the first significant viewport and is rewritten as If d min Not greater than the threshold d th , then traverse P2, and first obtain the transformed coordinate position P2' of P2 in the same way as the process for P1, then calculate the angular distance between the transformed coordinate position P2' and the coordinates of the spherical center point of N1 spliced viewports, and then obtain the minimum angular distance. Finally, if the minimum angular distance is greater than the threshold d th, then P2' is taken as the coordinate of the spherical center point of the first significant viewport and is rewritten as If the minimum angle distance is not greater than the threshold d th , then traverse P3, and so on, until the minimum angle distance is found to be greater than the threshold d th Under the condition of b , P b The transformed coordinate position P b 'As the coordinates of the spherical center point of the first significant viewport, re-recorded as Among them, min() is the minimum function, P b ∈{P1,P2,P3,...,P W×H}, P b =(u b ,v b ) represents the b-th coordinate position after arrangement, b∈[1,W×H], P2'=(φ2,θ2), P b '=(φ b ,θ b ), The longitude coordinate of the spherical center point of the first significant viewport, The latitude coordinate of the spherical center point of the first significant viewport;
[0033] Step 2_5: Select the coordinates of the spherical center points of the 2nd to N2th significant viewports in sequence. The coordinates of the spherical center point of the prominent viewport The specific process is:
[0034] Set the selected The coordinates of the spherical center point of the prominent viewport Corresponding to {P1,P2,P3,...,P W×H} Then in {P1,P2,P3,...,P W×H} Start traversing each subsequent coordinate position in turn, Convert from the equirectangular coordinate system to the geometric coordinate system to obtain the transformed coordinate position, recorded as Then calculate separately The angular distance from the coordinates of the spherical center point of the N1 stitching viewports, The coordinates of the spherical center point of the i-th stitching viewport The angular distance is recorded as Then get the minimum angle distance, recorded as And calculate separately With the selected 1st to The angular distance between the coordinates of the spherical center point of the significant viewport, With The angular distance between the coordinates of the spherical center points of the prominent viewports is recorded as Then get the minimum angle distance, recorded as Re-judge and Are they all greater than the threshold d? th ,if and Both are greater than the threshold d th , then As the The coordinates of the spherical center point of the prominent viewport are rewritten as If not satisfied and Both are greater than the threshold d th , then traverse According to the target The process is the same as first obtaining Transformed coordinate position Then calculate the transformed coordinate position The angular distance of the coordinates of the spherical center point of N1 stitching viewports and the minimum angular distance are obtained, and the transformed coordinate position is calculated The angular distance from the coordinates of the spherical center points of all selected significant viewports and the minimum angular distance are obtained. Finally, if both minimum angular distances are greater than the threshold d th , then As the The coordinates of the spherical center point of the prominent viewport are rewritten as If the two minimum angle distances obtained are not greater than the threshold d th , then traverse And so on, until the two minimum angular distances are found to be greater than the threshold d th The coordinate position under the condition of; where N2≥1, The initial value of is 2. Indicates The longitude coordinates of the spherical center points of the prominent viewports, Indicates The latitude coordinates of the spherical center points of the prominent viewports, Indicates the first Coordinate positions, Indicates the first Coordinate positions,
[0035] Step 2_6: The coordinates of the spherical center point of the first significant viewport obtained in step 2_4 The coordinates of the spherical center points of the 2nd to N2th significant viewports obtained in step 2_5 constitute a set, which is recorded as Then I sphere,S The coordinates of the spherical center points of the N2 significant viewports are projected to obtain N2 distorted significant viewport images. sphere,S The coordinates of the sphere center point along the jth salient viewport The jth distorted viewport image obtained by projection is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Among them, 1≤j≤N2, represents the coordinates of the spherical center point of the jth salient viewport, represents the longitude coordinate of the spherical center point of the jth significant viewport, represents the latitude coordinate of the spherical center point of the jth salient viewport, represents the longitude coordinate of the jth significant viewport at the spherical sampling point, represents the latitude coordinate of the jth significant viewport at the spherical sampling point, The step size is The step size is β j represents the jth distorted viewport image The viewing angle size;
[0036] Follow steps 2_1 to 2_6 to obtain I S The corresponding N1 distorted spliced viewport images and N2 distorted significant viewport images are obtained in the same way. G The corresponding N1 reference stitching viewport images and N2 reference salient viewport images are I G The corresponding i-th reference stitching viewport image is recorded as Will I G The corresponding j-th reference salient viewport image is recorded as
[0037] In the step 5, The acquisition process is:
[0038] Step 5_1: Convert to grayscale image, denoted as Then calculate The guiding matrix corresponding to each pixel in The steering matrix corresponding to the qth pixel in is recorded as in, express The total number of pixels in Ω q Indicates The coordinate position of the qth pixel in The set of coordinate positions of all pixels in the rectangular window with the center point as the center point, Indicates The coordinate position of the qth pixel in The coordinate position of the q'th pixel in the rectangular window with the center point as the center point, Indicates The coordinate position of the qth pixel in The total number of pixels in the rectangular window with the center point as the center point, express China-Israel The coordinate position of the qth pixel in is the pixel value of the q'th pixel in the rectangular window at the center point, Indicates The coordinate position of the qth pixel in The horizontal coordinate of the q'th pixel in the rectangular window with the center point as the center point, Indicates The coordinate position of the qth pixel in The ordinate of the q'th pixel in the rectangular window with the center point as the center point, symbol To find the sign of partial derivative;
[0039] Step 5_2: Calculation The similarity between each pixel in the p×p neighborhood around the central pixel is The qth pixel in the image and the qth pixel in the p×p neighborhood around it as the center pixel The similarity between neighboring pixels is recorded as Then get The local guide kernel feature vector of each pixel in The local guide kernel feature vector of the qth pixel in is recorded as Among them, p ≥ 3, exp() represents an exponential function with the natural base e as the base. express China-Israel The qth pixel in the p×p neighborhood around the central pixel The coordinate position of the neighboring pixel points, if exist Outside the scope The dimension of h is 1×(p×p-1), q,1 express The similarity between the qth pixel in the image and the first neighboring pixel in the p×p neighborhood around the center pixel, h q,2 express The similarity between the qth pixel in the image and the second neighboring pixel in the p×p neighborhood around the centered pixel, h q,p×p-1 express The similarity between the qth pixel in and the p×p-1th neighboring pixel in the p×p neighborhood around the centered pixel;
[0040] Step 5_3: Combination The local guide kernel feature vectors of all pixels in are obtained The local guide kernel feature tensor of Will The mid-coordinate position is The pixel value of the pixel point is recorded as Then principal component analysis was used to extract The first principal component, the second principal component, and the third principal component are recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as
[0041] Step 5_4: Follow the process from step 5_1 to step 5_3 to obtain The grayscale image is calculated by The guidance matrix corresponding to each pixel in the grayscale image is obtained The local guided kernel feature tensor of the grayscale image Re-extraction The first principal component, the second principal component, and the third principal component are recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as
[0042] Step 5_5: Calculation The mean pixel value of the ξth superpixel block in is denoted as Similarly, calculation The mean pixel value of the ξth superpixel block in is denoted as calculate The mean pixel value of the ξth superpixel block in is denoted as Calculated in Zhongyu The mean pixel value of the superpixel block matched by the ξth superpixel block in is denoted as Calculated in Zhongyu The mean pixel value of the superpixel block matched by the ξth superpixel block in is denoted as Calculated in Zhongyu The mean pixel value of the superpixel block matched by the ξth superpixel block in is denoted as in, Representation Statistics The total number of pixels in the ξth superpixel block in , Indicates that statistics Zhongyu The total number of pixels in the superpixel block matched by the ξ-th superpixel block in ;
[0043] Step 5_6: Calculate Zhongyu The edge-aware features of the superpixel block matched by the ξth superpixel block in Calculated in Zhongyu The edge-aware features of the superpixel block matched by the ξth superpixel block in Calculated in Zhongyu The edge-aware features of the superpixel block matched by the ξth superpixel block in
[0044] Step 5_7: Concatenate to form a feature vector, as
[0045] In step 6, The acquisition process is:
[0046] Step 6_1: Use aspect ratio similarity evaluation method to obtain The ξth superpixel block in Zhongyu The aspect ratio similarity of the superpixel block matched by the ξth superpixel block in is denoted as according to and Calculated;
[0047] Step 6_2: Extraction All boundary points in the ξ-th superpixel block in form a set, denoted as Similarly, extract Zhongyu All boundary points in the superpixel block matched by the ξth superpixel block in , and form a set, recorded as Then a dynamic time planning algorithm is used to obtain The ξth superpixel block in Zhongyu The boundary similarity of the superpixel block matched by the ξth superpixel block in is denoted as according to and Calculated;
[0048] Step 6_3: and Concatenate to form a feature vector, as
[0049] In the step 7, The acquisition process is:
[0050] Step 7_1: In the HSV color space, the H component, S component, and V component are recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as
[0051] Step 7_2: In the HSV color space, the H component, S component, and V component are recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The pixel value of the pixel point with coordinate position is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as
[0052] Step 7_3: Calculation The mean value of the hue information of the ξth superpixel block in is denoted as And calculated in Zhongyu The mean value of the hue information of the superpixel block matched by the ξth superpixel block in Then calculate Zhongyu The color perception feature of the H channel of the superpixel block matched by the ξth superpixel block in is recorded as
[0053] Similarly, calculation The mean saturation information of the ξth superpixel block in is denoted as And calculated in Zhongyu The mean saturation information of the superpixel block matched by the ξth superpixel block in is recorded as Then calculate Zhongyu The color perception feature of the S channel of the superpixel block matched by the ξth superpixel block in is recorded as
[0054] Similarly, calculation The mean brightness information of the ξth superpixel block in is denoted as And calculated in Zhongyu The mean brightness information of the superpixel block matched by the ξth superpixel block in is recorded as Then calculate Zhongyu The color perception feature of the V channel of the superpixel block matched by the ξth superpixel block in is recorded as
[0055] in, Representation Statistics The total number of pixels in the ξth superpixel block in , Indicates that statistics Zhongyu The total number of pixels in the superpixel block matched by the ξ-th superpixel block in ;
[0056] Step 7_3: Concatenate to form a feature vector, as
[0057] In the step 8, The acquisition process is:
[0058] Step 8_1: Calculation The structural complexity weight of each superpixel block in The structural complexity weight of the ξth superpixel block in is recorded as Among them, exp() represents the exponential function with the natural base e as the base, σ is the parameter used to control the saturation point of the nonlinear function, express The average edge strength of the ξth superpixel block in express The mean pixel value of the ξth superpixel block in express The mean pixel value of the ξth superpixel block in express The mean pixel value of the ξth superpixel block in express The first principal component is obtained by principal component analysis of the local guided kernel feature tensor of the grayscale image. express The second principal component is obtained by principal component analysis of the local guided kernel feature tensor of the grayscale image. express The third principal component is obtained by principal component analysis of the local guided kernel feature tensor of the grayscale image;
[0059] Step 8_2: Calculation The matching confidence weight of each superpixel block in The matching confidence weight of the ξth superpixel block in is recorded as in, Representation Statistics The total number of pixels in the ξth superpixel block in , δ is a constant used to improve stability, According to and The pixel matching relationship between The reference viewport image reconstructed from the matching pixels in The mid-coordinate position is The pixel value of the pixel point;
[0060] Step 8_3: Calculation The final weight of each superpixel block in The final weight of the ξ-th superpixel block in is recorded as Then get
[0061] Compared with the prior art, the advantages of the present invention are:
[0062] The method of the present invention combines the distortion distribution characteristics of panoramic stitching images and people's viewing behavior, predicts the stitching viewport image and the significant viewport image, in the training stage, considering the influence of distortions such as ghosting, deformation, chromatic aberration, and holes on image quality, extracts edge perception features, deformation perception features, color perception features, and information perception features to form a feature vector, and combines the stitching viewport feature vector and the significant viewport feature vector to form a feature vector of the panoramic stitching image, then uses multiple linear regression to train the feature vectors of all panoramic stitching images, and constructs a multiple linear regression training model of the panoramic stitching images; in the testing stage, by calculating the feature vectors of the panoramic stitching images in the testing set, and according to the multiple linear regression training model of the panoramic stitching images constructed in the training stage, predicts the objective quality evaluation prediction value of the panoramic stitching images in the testing set, because the obtained feature vector has strong stability and more comprehensively reflects the distortion of the panoramic stitching images, therefore, the correlation between the objective evaluation result and the subjective perception of the human eye is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] Figure 1 The figure is a general implementation block diagram of the method of the present invention. DETAILED DESCRIPTION
[0064] The present invention is further described in detail below with reference to the accompanying drawings.
[0065] The present invention proposes a method for evaluating the quality of panoramic stitching images, and its overall implementation block diagram is as follows: Figure 1 As shown, it includes the following steps:
[0066] Step 1: Select two panoramic stitching images obtained by stitching two fisheye images at different orientations, and use the panoramic stitching image obtained by stitching two fisheye images at 90° orientation and 270° orientation as the distorted image, and record it as I S , will I S The pixel value of the pixel point with coordinate position (u, v) is recorded as I S (u, v); The panoramic stitching image obtained by stitching the two fisheye images at 0° and 180° is used as the reference image and is recorded as I G , will I G The pixel value of the pixel point with coordinate position (u, v) is recorded as I G(u,v); where 1≤u≤W, 1≤v≤H, W represents I S and I G Width, H represents I S and I G height.
[0067] Step 2: Considering the distortion distribution characteristics of panoramic stitching images and people’s viewing behavior, S Perform projection transformation to obtain I S The corresponding N1 distorted spliced viewport images and N2 distorted significant viewport images are I S The corresponding i-th distorted spliced viewport image is recorded as Will I S The corresponding j-th distorted viewport image is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as
[0068] Similarly, for I G Perform projection transformation to obtain I G The corresponding N1 reference stitching viewport images and N2 reference salient viewport images are I G The corresponding i-th reference stitching viewport image is recorded as Will I G The corresponding j-th reference salient viewport image is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as
[0069] Among them, N1≥1, N2≥1, in this embodiment, N1=6, N2=6, 1≤i≤N1, 1≤j≤N2.
[0070] In this specific embodiment, in step 2, and and and The acquisition process is:
[0071] Step 2_1: I S Convert from the equirectangular coordinate system to the geometric coordinate system to obtain the corresponding spherical panoramic image, denoted as I sphere,S , will I sphere,SThe pixel value of the pixel point with the coordinate position (φ,θ) is recorded as I sphere,S (φ,θ), Among them, φ is the longitude coordinate in the geometric coordinate system, θ is the latitude coordinate in the geometric coordinate system, -π≤φ≤π, π=3.14159…, φ=0 and φ=π correspond to the positions of the two seams.
[0072] Step 2_2: I sphere,S The coordinates of the spherical center points of the N1 stitching viewports are projected to obtain N1 distorted stitching viewport images. sphere,S The coordinates of the center point of the sphere along the i-th stitching viewport The i-th distorted spliced viewport image obtained by projection is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Among them, N1≥1, 1≤i≤N1, Represents the coordinate position Right now Represents the longitude coordinate of the spherical center point of the i-th stitching viewport, Represents the latitude coordinate of the spherical center point of the i-th stitching viewport, Represents the longitude coordinate of the i-th stitching viewport at the spherical sampling point, Represents the latitude coordinate of the i-th stitching viewport at the spherical sampling point, The step size is The step size is α i Represents the i-th distorted stitched viewport image The viewing angle size.
[0073] In this embodiment, N1=6, The symbol “∪” is the union operator, which is the first distorted spliced viewport image. The view angle α1 and the 4th distorted stitched viewport image The viewing angle α4 is Second distorted stitched viewport image The third distorted spliced viewport image Viewing angle α3, the fifth distorted stitching viewport image Viewing angle α5 and the 6th distorted stitched viewport image The viewing angle α6 is
[0074] Step 2_3: Use the existing panoramic convolutions for 360° single-image saliency prediction method to extract I S The saliency map of A , will I A The pixel value of the pixel point with coordinate position (u, v) is I S The saliency value of the pixel with coordinate position (u, v) is recorded as I A (u,v); then for I A The coordinate positions of all pixels in are arranged in descending order according to the pixel values of the pixels, and the set of arranged coordinate positions is recorded as {P1, P2, P3, ..., P W×H}; where P1 = (u1, v1) represents the first coordinate position after arrangement, i.e. I A The coordinate position of the pixel with the largest pixel value in the array, P2 = (u2, v2) represents the second coordinate position after arrangement, which is I A The coordinate position of the pixel with the second largest pixel value, P3 = (u3, v3) represents the third coordinate position after arrangement, which is I A The coordinate position of the pixel with the third largest pixel value, P W×H =(u W×H ,v W×H ) represents the W×Hth coordinate position after arrangement, which is I A The coordinate position of the pixel with the smallest pixel value.
[0075] Step 2_4: Select the coordinates of the spherical center point of the first significant viewport The specific process is:
[0076] Traverse {P1,P2,P3,...,P W×H}, transform P1 from the equirectangular coordinate system to the geometric coordinate system to obtain the transformed coordinate position, recorded as P1', P1'=(φ1,θ1), Then calculate the angular distance between P1' and the coordinates of the spherical center point of N1 stitching viewports, and replace P1' with the coordinates of the spherical center point of the i-th stitching viewport The angular distance is denoted as d i , Then get the minimum angle distance, recorded as d min , d min =min(d i |1≤i≤N1); then judge d minIs it greater than the threshold d? th , if d min Greater than the threshold d th , then P1' is taken as the coordinate of the spherical center point of the first significant viewport and is rewritten as If d min Not greater than the threshold d th , then traverse P2, and first obtain the transformed coordinate position P2' of P2 in the same way as the process for P1, then calculate the angular distance between the transformed coordinate position P2' and the coordinates of the spherical center point of N1 spliced viewports, and then obtain the minimum angular distance. Finally, if the minimum angular distance is greater than the threshold d th , then P2' is taken as the coordinate of the spherical center point of the first significant viewport and is rewritten as If the minimum angle distance is not greater than the threshold d th , then traverse P3, and so on, until the minimum angle distance is found to be greater than the threshold d th Under the condition of b , P b The transformed coordinate position P b 'As the coordinates of the spherical center point of the first significant viewport, re-recorded as Wherein, min() is the minimum value function. In this embodiment, d th =30°, P b ∈{P1,P2,P3,...,P W×H}, P b =(u b ,v b ) represents the b-th coordinate position after arrangement, b∈[1,W×H], P2'=(φ2,θ2), P b '=(φ b ,θ b ), The longitude coordinate of the spherical center point of the first significant viewport, Represents the latitude coordinate of the spherical center point of the first significant viewport.
[0077] Step 2_5: Select the coordinates of the spherical center points of the 2nd to N2th significant viewports in sequence. The coordinates of the spherical center point of the prominent viewport The specific process is:
[0078] Set the selected The coordinates of the spherical center point of the prominent viewport Corresponding to {P1,P2,P3,...,P W×H} Then in {P1,P2,P3,...,P W×H} Start traversing each subsequent coordinate position in turn, Convert from the equirectangular coordinate system to the geometric coordinate system to obtain the transformed coordinate position, recorded as Then calculate separately The angular distance from the coordinates of the spherical center point of the N1 stitching viewports, The coordinates of the spherical center point of the i-th stitching viewport The angular distance is recorded as Then get the minimum angle distance, recorded as And calculate separately With the selected 1st to The angular distance between the coordinates of the spherical center point of the significant viewport, With The angular distance between the coordinates of the spherical center points of the prominent viewports is recorded as Then get the minimum angle distance, recorded as Re-judge and Are they all greater than the threshold d? th ,if and Both are greater than the threshold d th , then As the The coordinates of the spherical center point of the prominent viewport are rewritten as If not satisfied and Both are greater than the threshold d th , then traverse According to the target The process is the same as first obtaining Transformed coordinate position Then calculate the transformed coordinate position The angular distance of the coordinates of the spherical center point of N1 stitching viewports and the minimum angular distance are obtained, and the transformed coordinate position is calculated The angular distance from the coordinates of the spherical center points of all selected significant viewports and the minimum angular distance are obtained. Finally, if both minimum angular distances are greater than the threshold d th , then As the The coordinates of the spherical center point of the prominent viewport are rewritten as If the two minimum angle distances obtained are not greater than the threshold d th , then traverse And so on, until the two minimum angular distances are found to be greater than the threshold d th The coordinate position under the condition of; wherein, N2≥1, in this embodiment, N2=6, The initial value of is 2. Indicates The longitude coordinates of the spherical center points of the prominent viewports, Indicates The latitude coordinates of the spherical center points of the prominent viewports, Indicates the first Coordinate positions, Indicates the first Coordinate positions,
[0079] Step 2_6: The coordinates of the spherical center point of the first significant viewport obtained in step 2_4 The coordinates of the spherical center points of the 2nd to N2th significant viewports obtained in step 2_5 constitute a set, which is recorded as Then I sphere,S The coordinates of the spherical center points of the N2 significant viewports are projected to obtain N2 distorted significant viewport images. sphere,S The coordinates of the sphere center point along the jth salient viewport The jth distorted viewport image obtained by projection is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Among them, 1≤j≤N2, represents the coordinates of the spherical center point of the jth salient viewport, represents the longitude coordinate of the spherical center point of the jth significant viewport, represents the latitude coordinate of the spherical center point of the jth salient viewport, represents the longitude coordinate of the jth significant viewport at the spherical sampling point, represents the latitude coordinate of the jth significant viewport at the spherical sampling point, The step size is The step size is β j represents the jth distorted viewport image In this embodiment,
[0080] Follow steps 2_1 to 2_6 to obtain I SThe corresponding N1 distorted spliced viewport images and N2 distorted significant viewport images are obtained in the same way. G The corresponding N1 reference stitching viewport images and N2 reference salient viewport images are I G The corresponding i-th reference stitching viewport image is recorded as Will I G The corresponding j-th reference salient viewport image is recorded as
[0081] Step 3: Use the existing SIFT-Flow method to establish I G Each corresponding reference stitching viewport image and I S The matching relationship between each corresponding distorted spliced viewport image is obtained. G The corresponding SIFT-Flow vector of each pixel in each reference stitched viewport image is and Using the existing SIFT-Flow method to establish and The matching relationship between The SIFT-Flow vector of each pixel in The mid-coordinate position is The SIFT-Flow vector of the pixel point is recorded as
[0082] Similarly, the existing SIFT-Flow method is used to establish I G Each corresponding reference salient viewport image is S The matching relationship between each distorted viewport image is obtained by G The corresponding SIFT-Flow vector of each pixel in each reference salient viewport image is and Using the existing SIFT-Flow method to establish and The matching relationship between The SIFT-Flow vector of each pixel in The mid-coordinate position is The SIFT-Flow vector of the pixel point is recorded as
[0083] in, Used to indicate the horizontal direction. Used to indicate the vertical direction. express The horizontal offset of express The vertical offset of express The horizontal offset of express The vertical offset of the .
[0084] Step 4: Use the existing Simple Linear Iterative Clustering (SLIC) method to cluster I G Each corresponding reference spliced viewport image is divided into superpixels. right Perform superpixel block division to obtain N seed non-overlapping superpixel blocks, The set of coordinate positions of all pixels in the ξth superpixel block is recorded as Then according to I G The corresponding SIFT-Flow vector of each pixel in each reference stitching viewport image is determined by I G Each superpixel block in each corresponding reference stitching viewport image is in I S The corresponding superpixel blocks in each distorted spliced viewport image are according to The SIFT-Flow vector of each pixel in is determined in Zhongyu Each superpixel block in the matching superpixel block will be Zhongyu The set of coordinate positions of all pixels in the superpixel block matched by the ξth superpixel block in is recorded as Among them, N seed ≥1, in this embodiment, N seed =50,1≤ξ≤N seed .
[0085] Similarly, the existing Simple Linear Iterative Clustering (SLIC) method is used to cluster I G Each corresponding reference salient viewport image is divided into superpixels. right Perform superpixel block division to obtain N seed non-overlapping superpixel blocks, The set of coordinate positions of all pixels in the ξth superpixel block is recorded as Then according to I G The corresponding SIFT-Flow vector of each pixel in each reference salient viewport image is determined by I GEach superpixel block in each corresponding reference salient viewport image is in I S The corresponding superpixel blocks in each distorted viewport image are according to The SIFT-Flow vector of each pixel in is determined in Zhongyu Each superpixel block in the matching superpixel block will be Zhongyu The set of coordinate positions of all pixels in the superpixel block matched by the ξth superpixel block in is recorded as
[0086] Step 5: According to I G Each superpixel block and I in the first principal component, the second principal component, and the third principal component of the local guided kernel feature tensor of the grayscale image of each reference stitching viewport image G Each superpixel block in the first principal component, the second principal component, and the third principal component of the local guided kernel feature tensor of the grayscale image of each corresponding reference spliced viewport image is in I S The corresponding superpixel blocks in the first principal component, the second principal component, and the third principal component of the local guided kernel feature tensor of the grayscale image of each distorted spliced viewport image are calculated. S The edge-aware features of each superpixel block in each distorted spliced viewport image will be Zhongyu The edge-aware feature of the superpixel block matched by the ξth superpixel block in is recorded as in, The dimension of is 1×3, Indicated in The first principal component of the local guided kernel feature tensor of the grayscale image Zhongyu The first principal component of the local guided kernel feature tensor of the grayscale image The edge-aware features of the superpixel block matched by the ξ-th superpixel block in Indicated in The second principal component of the local guided kernel feature tensor of the grayscale image Zhongyu The second principal component of the local guided kernel feature tensor of the grayscale image The edge-aware features of the superpixel block matched by the ξ-th superpixel block in Indicated in The third principal component of the local guided kernel feature tensor of the grayscale image Zhongyu The third principal component of the local guided kernel feature tensor of the grayscale image The edge-aware features of the superpixel patch matched to the ξ-th superpixel patch in .
[0087] In this specific embodiment, in step 5, The acquisition process is:
[0088] Step 5_1: Convert to grayscale image, denoted as Then calculate The guiding matrix corresponding to each pixel in The steering matrix corresponding to the qth pixel in is recorded as in, express The total number of pixels in Ω q Indicates The coordinate position of the qth pixel in The coordinate positions of all pixels in the rectangular window with the center point as the set. In this embodiment, the size of the rectangular window is 5×5. Indicates The coordinate position of the qth pixel in The coordinate position of the q'th pixel in the rectangular window with the center point as the center point, Indicates The coordinate position of the qth pixel in The total number of pixels in the rectangular window with the center point is express China-Israel The coordinate position of the qth pixel in is the pixel value of the q'th pixel in the rectangular window at the center point, Indicates The coordinate position of the qth pixel in The horizontal coordinate of the q'th pixel in the rectangular window with the center point as the center point, Indicates The coordinate position of the qth pixel in The ordinate of the q'th pixel in the rectangular window with the center point as the center point, symbol To find the sign of the partial derivative.
[0089] Here you can The steering matrix corresponding to each pixel in is spectrally decomposed to obtain two singular values. The process of spectral decomposition is: The two singular values obtained are recorded as s1 and s2 respectively. Then, while keeping the eigenvector of the main direction unchanged, the two singular values are regularized to transform the spectral decomposition formula of the steering matrix into the Riemann metric, which is described as: Among them, λ1 and λ2 are the two eigenvalues obtained by spectral decomposition, and the superscript "T" represents the transpose of a vector or matrix. and express The corresponding main direction eigenvector, ε and τ are constants used to increase stability, Ψ is a constant used to enhance or suppress local gradient information, in this embodiment, ε=10 -1 ,τ=1,Ψ=1.
[0090] Step 5_2: Calculation The similarity between each pixel in the p×p neighborhood around the central pixel is The qth pixel in the image and the qth pixel in the p×p neighborhood around it as the center pixel The similarity between neighboring pixels is recorded as Then get The local steering kernel (LSK) feature vector of each pixel in The local guide kernel feature vector of the qth pixel in is recorded as Wherein, p≥3, in this embodiment, p=5, exp() represents the exponential function with the natural base e as the base, e=2.71828..., express China-Israel The qth pixel in the p×p neighborhood around the central pixel The coordinate position of the neighboring pixel points, if exist Outside the scope The dimension of h is 1×(p×p-1), q,1 express The similarity between the qth pixel in the image and the first neighboring pixel in the p×p neighborhood around the center pixel, h q,2 express The similarity between the qth pixel in the image and the second neighboring pixel in the p×p neighborhood around the centered pixel, h q,p×p-1 express The similarity between the qth pixel in the image and the p×p-1th neighboring pixel in the p×p neighborhood around the pixel as the center is The qth pixel in is the p×p neighborhood around the central pixel, and the p×p-1 pixels excluding the central pixel are called the neighborhood pixels of the central pixel.
[0091] Step 5_3: Combination The local guide kernel feature vectors of all pixels in are obtained The local guide kernel feature tensor of Will The mid-coordinate position is The pixel value of the pixel point is recorded as Then the existing principal component analysis (PCA) method is used to extract The first principal component, the second principal component, and the third principal component are recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as
[0092] Step 5_4: Follow the process from step 5_1 to step 5_3 to obtain The grayscale image is calculated by The guidance matrix corresponding to each pixel in the grayscale image is obtained The local guided kernel feature tensor of the grayscale image Re-extraction The first principal component, the second principal component, and the third principal component are recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as
[0093] Step 5_5: Calculation The mean pixel value of the ξth superpixel block in is denoted as Similarly, calculation The mean pixel value of the ξth superpixel block in is denoted as calculate The mean pixel value of the ξth superpixel block in is denoted as Calculated in Zhongyu The mean pixel value of the superpixel block matched by the ξth superpixel block in is denoted as Calculated in Zhongyu The mean pixel value of the superpixel block matched by the ξth superpixel block in is denoted as Calculated in Zhongyu The mean pixel value of the superpixel block matched by the ξth superpixel block in is denoted as in, Representation Statistics The total number of pixels in the ξth superpixel block in , Indicates that statistics Zhongyu The total number of pixels in the superpixel block that matches the ξth superpixel block in .
[0094] Step 5_6: Calculate Zhongyu The edge-aware features of the superpixel block matched by the ξth superpixel block in Calculated in Zhongyu The edge-aware features of the superpixel block matched by the ξth superpixel block in Calculated in Zhongyu The edge-aware features of the superpixel block matched by the ξth superpixel block in
[0095] Step 5_7: Concatenate to form a feature vector, as
[0096] Step 6: According to I G Each superpixel block and I in each corresponding reference spliced viewport image G Each superpixel block in each corresponding reference stitching viewport image is in I S The corresponding superpixel blocks in each distorted spliced viewport image are calculated S The deformation-aware features of each superpixel block in each distorted spliced viewport image will be Zhongyu The deformation-aware feature of the superpixel block matched by the ξth superpixel block in is recorded as in, The dimension of is 1×2, express The ξth superpixel block in Zhongyu The aspect ratio similarity of the superpixel block matched by the ξ-th superpixel block in , express The ξth superpixel block in Zhongyu The boundary similarity of the superpixel block matched by the ξ-th superpixel block in .
[0097] In this specific embodiment, in step 6, The acquisition process is:
[0098] Step 6_1: Use the existing aspect ratio similarity evaluation method to obtain The ξth superpixel block in Zhongyu The aspect ratio similarity of the superpixel block matched by the ξth superpixel block in is denoted as according to and Calculated.
[0099] Step 6_2: Extraction All boundary points in the ξ-th superpixel block (i.e., the pixels on the contour line) in the ξ-th superpixel block ... Similarly, extract Zhongyu All boundary points in the superpixel block matched by the ξth superpixel block in , and form a set, recorded as Then, the existing dynamic time planning algorithm (Dynamic TimeWarping, DTW) is used to obtain The ξth superpixel block in Zhongyu The boundary similarity of the superpixel block matched by the ξth superpixel block in is denoted as according to and Calculated.
[0100] Step 6_3: and Concatenate to form a feature vector, as
[0101] Step 7: According to I G Each superpixel block and I in the H component, S component and V component of each corresponding reference spliced viewport image GEach superpixel block in the H component, S component, and V component of each corresponding reference stitching viewport image is in I S The corresponding superpixel blocks in the H component, S component, and V component of each distorted spliced viewport image are calculated. S The color perception features of each superpixel block in each distorted spliced viewport image will be Zhongyu The color perception feature of the superpixel block matched by the ξth superpixel block in is recorded as in, The dimension of is 1×3, Indicated in Zhongyu The color perception feature of the H channel of the superpixel block matched by the ξth superpixel block in Indicated in Zhongyu The color perception feature of the S channel of the superpixel block matched by the ξth superpixel block in express Zhongyu The color perception features of the V channel of the superpixel block matched by the ξ-th superpixel block in .
[0102] In this specific embodiment, in step 7, The acquisition process is:
[0103] Step 7_1: In the HSV color space, the H component, S component, and V component are recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as
[0104] Step 7_2: In the HSV color space, the H component, S component, and V component are recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The pixel value of the pixel point with coordinate position is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as
[0105] Step 7_3: Calculation The mean value of the hue information of the ξth superpixel block in is denoted as And calculated in Zhongyu The mean value of the hue information of the superpixel block matched by the ξth superpixel block in Then calculate Zhongyu The color perception feature of the H channel of the superpixel block matched by the ξth superpixel block in is recorded as
[0106] Similarly, calculation The mean saturation information of the ξth superpixel block in is denoted as And calculated in Zhongyu The mean saturation information of the superpixel block matched by the ξth superpixel block in is recorded as Then calculate Zhongyu The color perception feature of the S channel of the superpixel block matched by the ξth superpixel block in is recorded as
[0107] Similarly, calculation The mean brightness information of the ξth superpixel block in is denoted as And calculated in Zhongyu The mean brightness information of the superpixel block matched by the ξth superpixel block in is recorded as Then calculate Zhongyu The color perception feature of the V channel of the superpixel block matched by the ξth superpixel block in is recorded as
[0108] in, Representation Statistics The total number of pixels in the ξth superpixel block in , Indicates that statistics Zhongyu The total number of pixels in the superpixel block that matches the ξth superpixel block in .
[0109] Step 7_3: Concatenate to form a feature vector, as
[0110] Step 8: I S The edge-aware features of all superpixel blocks in each corresponding distorted spliced viewport image are adaptively fused to obtain I S The corresponding edge-perceived features of each distorted spliced viewport image are The edge perception feature is recorded as Similarly, for I S The deformation perception features of all superpixel blocks in each distorted spliced viewport image are adaptively fused to obtain I S The corresponding deformation perception features of each distorted spliced viewport image are The deformation perception feature is recorded as to I S The color perception features of all superpixel blocks in each distorted spliced viewport image are adaptively fused to obtain I S The corresponding color perception features of each distorted spliced viewport image are The color perception characteristics of in, The dimension of is 1×3, The dimension of is 1×2, The dimension is 1×3.
[0111] In this specific embodiment, in step 8, The acquisition process is:
[0112] Step 8_1: Calculation The structural complexity weight of each superpixel block in The structural complexity weight of the ξth superpixel block in is recorded as Wherein, exp() represents an exponential function with the natural base e as the base, e=2.71828..., σ is a parameter for controlling the saturation point of the nonlinear function, and in this embodiment, σ=0.1, express The average edge strength of the ξth superpixel block in express The mean pixel value of the ξth superpixel block in express The mean pixel value of the ξth superpixel block in express The mean pixel value of the ξth superpixel block in express The first principal component is obtained by principal component analysis of the local guided kernel feature tensor of the grayscale image. express The second principal component is obtained by principal component analysis of the local guided kernel feature tensor of the grayscale image. express The third principal component is obtained after principal component analysis of the local guided kernel feature tensor of the grayscale image; in order to obtain a better weighting effect, the average edge strength of the superpixel block is nonlinearly transformed.
[0113] Step 8_2: Calculation The matching confidence weight of each superpixel block in The matching confidence weight of the ξth superpixel block in is recorded as in, Representation Statistics The total number of pixels in the ξth superpixel block in the ξth superpixel block, δ is a constant used to improve stability, and in this embodiment, δ=10 -6 , Indicates based on and The pixel matching relationship between The reference viewport image reconstructed from the matching pixels in The mid-coordinate position is The pixel value of the pixel.
[0114] Step 8_3: Calculation The final weight of each superpixel block in The final weight of the ξ-th superpixel block in is recorded as Then get
[0115] Step 9: Pass I S Each corresponding distorted stitched viewport image and I G The mutual information value between each corresponding reference spliced viewport image is calculated S The corresponding information perception features of each distorted spliced viewport image are The information perception feature of in, express The information entropy of express The information entropy of express and The information entropy of the joint distribution, The dimension of is 1×1.
[0116] Step 10: According to I S The corresponding edge perception features, deformation perception features, color perception features, and information perception features of each distorted spliced viewport image are obtained. S The corresponding feature vector of each distorted spliced viewport image is The characteristic vector of in, Indicates that are concatenated to form a feature vector, The dimension is 1×9.
[0117] Step 11: Follow the process from step 5 to step 10 to obtain I in the same way. S The edge perception features, deformation perception features, and color perception features of each superpixel block in each distorted viewport image are obtained, thereby obtaining I S The corresponding edge perception features, deformation perception features, and color perception features of each distorted viewport image are obtained, and I S The corresponding information perception features of each distorted viewport image are finally obtained. S The corresponding feature vector of each distorted viewport image is The characteristic vector of in, express The edge perception features of The dimension of is 1×3, express The deformation perception characteristics of The dimension of is 1×2, express The color perception characteristics of The dimension of is 1×3, express Information perception characteristics, The dimension of is 1×1, The dimension of is 1 × 9. Here, the relevant information of the distorted salient viewport image and the reference salient viewport image is used, such as The SIFT-Flow vector of each pixel in The superpixel blocks in Zhongyu Each superpixel block in is matched to the superpixel block.
[0118] Step 12: According to I S The corresponding feature vectors and I of all distorted spliced viewport images S The corresponding feature vectors of all distorted viewport images are calculated. S The characteristic vector of is denoted as F, F = w T ×F T +w A ×F A ; Among them, F T represents the feature vector of the splicing area, F T The dimension of is 1×9, F Arepresents the salient region feature vector, F A The dimension of is 1×9, The dimension of F is 1×9, w T F T The weight, w A F A The weight, w T +w A =1, in this embodiment, w T =0.83, w A =0.17, In this embodiment,
[0119] Step 13: Select n' panoramic stitching images as distorted images and the cross-reference images corresponding to each panoramic stitching image as reference images to form an image set; then use the subjective quality evaluation method to obtain the average subjective score value of each distorted image in the image set, and record the average subjective score value of the i'th distorted image in the image set as MOS i' ; and according to the process of step 2 to step 12, the feature vector of each distorted image in the image set is obtained in the same way, and the feature vector of the i'th distorted image in the image set is recorded as F i' ; Then classify all distorted images in the image set according to the scene content, assuming that there are k' image classes; where n'>1, 1≤i'≤n', MOS i' ∈[0,7],F i' The dimension is 1×9, k′>1, and in this embodiment, k′=12.
[0120] Step 14: Randomly select all distorted images of k'-1 image classes from the image set to form a training set; then the feature vectors and average subjective rating values of all distorted images in the training set form a training sample data set; then use multivariate linear regression as a machine learning method to train all feature vectors in the training sample data set, so that the error between the regression function value obtained after training and the average subjective rating value is minimized, and the optimal weight vector is fitted. and the optimal bias term Reuse the optimal weight vector and the optimal bias term Construct a multivariate linear regression training model for panoramic stitching images, denoted as The total number of distorted images in the training set is m, 1≤m<n', is a function representation, the superscript "T" indicates the transpose of a vector or matrix, F inprepresents the input vector of the multivariate linear regression training model of the panoramic stitching image, is the feature vector of the distorted image, and F inp The dimension of is 1×9, The input vector F represents the multivariate linear regression training model of the panoramic stitching image. inp A linear function of .
[0121] Step 15: All distorted images of the remaining 1 image class in the image set constitute a test set; then the feature vectors of all distorted images in the test set constitute a test sample data set; then, according to the constructed multivariate linear regression training model of the panoramic stitching image, each feature vector in the test sample data set is tested to predict the objective quality evaluation prediction value of the distorted image corresponding to each feature vector in the test sample data set, and the objective quality evaluation prediction value of the distorted image corresponding to the nth feature vector in the test sample data set is recorded as quality η , Among them, 1≤η≤n'-m, F η represents the nth feature vector in the test sample data set, F η The dimension of is 1×9, Indicates F η A linear function of .
[0122] Step 16: Repeat steps 14 to 15 k' times in total, so that each distorted image in the image set belongs to the test set only once, and obtain the objective quality evaluation prediction value of each distorted image in the image set.
[0123] In order to further illustrate the feasibility and effectiveness of the method of the present invention, the method of the present invention was tested.
[0124] In this embodiment, the method of the present invention is used to test the cross-reference panoramic stitching image quality evaluation database established by Beijing University of Aeronautics and Astronautics. The database contains 12 different scenes, which can be divided into two categories: indoor and outdoor, with a total of 292 groups of fisheye images. Among them, indoor scenes include: conference rooms, classrooms, stairs, underground parking lots, dance rooms, reading rooms, lounges and corridors; outdoor scenes include: streets, basketball courts, residential areas and open areas. These fisheye images are stitched using 7 different stitching methods to obtain 2044 panoramic stitching images of different qualities, and the average subjective score value of each panoramic stitching image is given.
[0125] In this embodiment, three commonly used objective parameters for evaluating image quality evaluation methods are used as evaluation indicators, namely, Pearson linear correlation coefficient (PLCC), Spearman rank order correlation coefficient (SROCC), and root mean squared error (RMSE) under nonlinear regression conditions. PLCC and RMSE reflect the accuracy of the objective quality evaluation prediction value, and SROCC reflects its monotonicity. The Pearson correlation coefficient under nonlinear regression conditions is calculated for the objective quality evaluation prediction value and the average subjective rating value of each distorted image in the image set. The larger the value of the Pearson correlation coefficient under nonlinear regression conditions, the better the multivariate linear regression training model of the panoramic stitching image is; the Spearman correlation coefficient under nonlinear regression conditions is calculated for the objective quality evaluation prediction value and the average subjective rating value of each distorted image in the image set. The larger the value of the Spearman correlation coefficient under nonlinear regression conditions, the better the multivariate linear regression training model of the panoramic stitching image is; the mean square error under nonlinear regression conditions is calculated for the objective quality evaluation prediction value and the average subjective rating value of each distorted image in the image set. The smaller the value of the mean square error under nonlinear regression conditions, the better the multivariate linear regression training model of the panoramic stitching image is.
[0126] Table 1 shows the correlation between the objective quality evaluation prediction value and the average subjective score value obtained by the method of the present invention.
[0127] Table 1 Comparison of Pearson correlation coefficient, Spearman correlation coefficient and mean square error between the objective quality evaluation prediction value and the average subjective score value obtained by the method of the present invention
[0128] Pearson correlation coefficient Spearman correlation coefficient Mean square error Method of the present invention 0.8668 0.8298 0.8976
[0129] It can be seen from Table 1 that the correlation between the objective quality evaluation prediction value obtained by the method of the present invention and the average subjective score value is very high, which fully demonstrates that the objective evaluation result of the method of the present invention is consistent with the result of subjective perception of the human eye, which is sufficient to illustrate the feasibility and effectiveness of the method of the present invention.
Claims
1. A method for evaluating the quality of panoramic stitching images. Features The following steps are involved: Step 1: Select two panoramic stitching images obtained by stitching two fisheye images at different orientations, and use the panoramic stitching image obtained by stitching two fisheye images at 90° orientation and 270° orientation as the distorted image, and record it as I S , will I S The pixel value of the pixel point with coordinate position (u, v) is recorded as I S (u, v); The panoramic stitching image obtained by stitching the two fisheye images at 0° and 180° is used as the reference image and is recorded as I G , will I G The pixel value of the pixel point with coordinate position (u, v) is recorded as I G (u,v); where 1≤u≤W, 1≤v≤H, W represents I S and I G Width, H represents I S and I G Height; Step 2: I S Perform projection transformation to obtain I S The corresponding N 1 distorted stitched viewport images and N 2 A distorted viewport image, I S The corresponding i-th distorted spliced viewport image is recorded as Will I S The corresponding j-th distorted viewport image is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Similarly, for I G Perform projection transformation to obtain I G The corresponding N 1 reference stitched viewport images and N 2 A reference salient viewport image, I G The corresponding i-th reference stitching viewport image is recorded as Will I G The corresponding j-th reference salient viewport image is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Among them, N 1 ≥1, N 2 ≥1, 1≤i≤N 1 , 1≤j≤N 2 ; Step 3: Use SIFT-Flow method to establish I G Each corresponding reference stitching viewport image and I S The matching relationship between each corresponding distorted spliced viewport image is obtained. G The corresponding SIFT-Flow vector of each pixel in each reference stitched viewport image is and Using SIFT-Flow method to build and The matching relationship between The SIFT-Flow vector of each pixel in The mid-coordinate position is The SIFT-Flow vector of the pixel point is recorded as Similarly, SIFT-Flow method is used to establish I G Each corresponding reference salient viewport image is S The matching relationship between each distorted viewport image is obtained by G The corresponding SIFT-Flow vector of each pixel in each reference salient viewport image is and Using SIFT-Flow method to build and The matching relationship between The SIFT-Flow vector of each pixel in The mid-coordinate position is The SIFT-Flow vector of the pixel point is recorded as in, Used to indicate the horizontal direction. Used to indicate the vertical direction. express The horizontal offset of express The vertical offset of express The horizontal offset of express The vertical offset of Step 4: Use simple linear iterative clustering method to G Each corresponding reference spliced viewport image is divided into superpixels. right Perform superpixel block division to obtain N seed non-overlapping superpixel blocks, The set of coordinate positions of all pixels in the ξth superpixel block is recorded as Then according to I G The corresponding SIFT-Flow vector of each pixel in each reference stitching viewport image is determined by I G Each superpixel block in each corresponding reference stitching viewport image is in I S The corresponding superpixel blocks in each distorted spliced viewport image are according to The SIFT-Flow vector of each pixel in is determined in Zhongyu Each superpixel block in the matching superpixel block will be Zhongyu The set of coordinate positions of all pixels in the superpixel block matched by the ξth superpixel block in is recorded as Among them, N seed ≥1, 1≤ξ≤N seed ; Similarly, a simple linear iterative clustering method is used to G Each corresponding reference salient viewport image is divided into superpixels. right Perform superpixel block division to obtain N seed non-overlapping superpixel blocks, The set of coordinate positions of all pixels in the ξth superpixel block is recorded as Then according to I G The SIFT-Flow vector of each pixel in each reference salient viewport image is determined by G Each superpixel block in each corresponding reference salient viewport image is in I S The corresponding superpixel blocks in each distorted viewport image are according to The SIFT-Flow vector of each pixel in is determined in Zhongyu Each superpixel block in the matching superpixel block will be Zhongyu The set of coordinate positions of all pixels in the superpixel block matched by the ξth superpixel block in is recorded as Step 5: According to I G Each superpixel block and I in the first principal component, the second principal component, and the third principal component of the local guided kernel feature tensor of the grayscale image of each reference stitching viewport image G Each superpixel block in the first principal component, the second principal component, and the third principal component of the local guided kernel feature tensor of the grayscale image of each corresponding reference spliced viewport image is in I S The corresponding superpixel blocks in the first principal component, the second principal component, and the third principal component of the local guided kernel feature tensor of the grayscale image of each distorted spliced viewport image are calculated. S The edge-aware features of each superpixel block in each distorted spliced viewport image will be Zhongyu The edge-aware feature of the superpixel block matched by the ξth superpixel block in is recorded as in, The dimension of is 1×3, Indicated in The first principal component of the local guided kernel feature tensor of the grayscale image Zhongyu The first principal component of the local guided kernel feature tensor of the grayscale image The edge-aware features of the superpixel block matched by the ξ-th superpixel block in Indicated in The second principal component of the local guided kernel feature tensor of the grayscale image Zhongyu The second principal component of the local guided kernel feature tensor of the grayscale image The edge-aware features of the superpixel block matched by the ξ-th superpixel block in Indicated in The third principal component of the local guided kernel feature tensor of the grayscale image Zhongyu The third principal component of the local guided kernel feature tensor of the grayscale image The edge-aware features of the superpixel block matched by the ξ-th superpixel block in ; Step 6: According to I G Each superpixel block and I in each corresponding reference spliced viewport image G Each superpixel block in each corresponding reference stitching viewport image is in I S The corresponding superpixel blocks in each distorted spliced viewport image are calculated S The deformation-aware features of each superpixel block in each distorted spliced viewport image will be Zhongyu The deformation-aware feature of the superpixel block matched by the ξth superpixel block in is recorded as in, The dimension of is 1×2, express The ξth superpixel block in Zhongyu The aspect ratio similarity of the superpixel block matched by the ξ-th superpixel block in , express The ξth superpixel block in Zhongyu The boundary similarity of the superpixel block matched by the ξ-th superpixel block in ; Step 7: According to I G Each superpixel block and I in the H component, S component and V component of each corresponding reference spliced viewport image G Each superpixel block in the H component, S component, and V component of each corresponding reference stitching viewport image is in I S The corresponding superpixel blocks in the H component, S component, and V component of each distorted spliced viewport image are calculated. S The color perception features of each superpixel block in each distorted spliced viewport image will be Zhongyu The color perception feature of the superpixel block matched by the ξth superpixel block in is recorded as in, The dimension of is 1×3, Indicated in Zhongyu The color perception feature of the H channel of the superpixel block matched by the ξth superpixel block in Indicated in Zhongyu The color perception feature of the S channel of the superpixel block matched by the ξth superpixel block in express Zhongyu The color perception feature of the V channel of the superpixel block matched by the ξ-th superpixel block in ; Step 8: I S The edge-aware features of all superpixel blocks in each corresponding distorted spliced viewport image are adaptively fused to obtain I S The corresponding edge-perceived features of each distorted spliced viewport image are The edge perception feature is recorded as Similarly, for I S The deformation perception features of all superpixel blocks in each distorted spliced viewport image are adaptively fused to obtain I S The corresponding deformation perception features of each distorted spliced viewport image are The deformation perception feature is recorded as to I S The color perception features of all superpixel blocks in each distorted spliced viewport image are adaptively fused to obtain I S The corresponding color perception features of each distorted spliced viewport image are The color perception characteristics of in, The dimension of is 1×3, The dimension of is 1×2, The dimension is 1×3; Step 9: Pass I S Each corresponding distorted stitched viewport image and I G The mutual information value between each corresponding reference spliced viewport image is calculated S The corresponding information perception features of each distorted spliced viewport image are The information perception feature is recorded as in, express The information entropy of express The information entropy of express and The information entropy of the joint distribution, The dimension of is 1×1; Step 10: According to I S The corresponding edge perception features, deformation perception features, color perception features, and information perception features of each distorted spliced viewport image are obtained. S The corresponding feature vector of each distorted spliced viewport image is The characteristic vector of in, Indicates that are concatenated to form a feature vector, The dimension is 1×9; Step 11: Follow the process from step 5 to step 10 to obtain I in the same way. S The edge perception features, deformation perception features, and color perception features of each superpixel block in each distorted viewport image are obtained, thereby obtaining I S The corresponding edge perception features, deformation perception features, and color perception features of each distorted viewport image are obtained, and I S The corresponding information perception features of each distorted viewport image are finally obtained. S The corresponding feature vector of each distorted viewport image is The characteristic vector of in, express The edge perception features of The dimension of is 1×3, express The deformation perception characteristics of The dimension of is 1×2, express The color perception characteristics of The dimension of is 1×3, express Information perception characteristics, The dimension of is 1×1, The dimension is 1×9; Step 12: According to I S The corresponding feature vectors and I of all distorted spliced viewport images S The corresponding feature vectors of all distorted viewport images are calculated. S The characteristic vector of is denoted as F, F = w T ×F T +w A ×F A ; Among them, F T represents the feature vector of the splicing area, F T The dimension of is 1×9, F A represents the salient region feature vector, F A The dimension of is 1×9, The dimension of F is 1×9, w T F T The weight, w A F A The weight, w T +w A =1, Step 13: Select n' panoramic stitching images as distorted images and the cross-reference images corresponding to each panoramic stitching image as reference images to form an image set; then use the subjective quality evaluation method to obtain the average subjective score value of each distorted image in the image set, and record the average subjective score value of the i'th distorted image in the image set as MOS i' ; and according to the process of step 2 to step 12, the feature vector of each distorted image in the image set is obtained in the same way, and the feature vector of the i'th distorted image in the image set is recorded as F i' ; Then classify all distorted images in the image set according to the scene content, assuming that there are k' image classes in total; where n'>1, 1≤i'≤n', MOS i' ∈[0,7],F i' The dimension is 1×9, k'>1; Step 14: Randomly select all distorted images of k'-1 image classes from the image set to form a training set; then the feature vectors and average subjective rating values of all distorted images in the training set form a training sample data set; then use multivariate linear regression as a machine learning method to train all feature vectors in the training sample data set, so that the error between the regression function value obtained after training and the average subjective rating value is minimized, and the optimal weight vector is fitted. and the optimal bias term Reuse the optimal weight vector and the optimal bias term Construct a multivariate linear regression training model for panoramic stitching images, denoted as The total number of distorted images in the training set is m, 1≤m<n', is a function representation, the superscript "T" indicates the transpose of a vector or matrix, F inp represents the input vector of the multivariate linear regression training model of the panoramic stitching image, is the feature vector of the distorted image, and F inp The dimension of is 1×9, The input vector F represents the multivariate linear regression training model of the panoramic stitching image. inp A linear function of Step 15: All distorted images of the remaining 1 image class in the image set constitute a test set; then the feature vectors of all distorted images in the test set constitute a test sample data set; then, according to the constructed multivariate linear regression training model of the panoramic stitching image, each feature vector in the test sample data set is tested to predict the objective quality evaluation prediction value of the distorted image corresponding to each feature vector in the test sample data set, and the objective quality evaluation prediction value of the distorted image corresponding to the nth feature vector in the test sample data set is recorded as Among them, 1≤η≤n'-m, F η represents the nth feature vector in the test sample data set, F η The dimension of is 1×9, Indicates F η A linear function of Step 16: Repeat steps 14 to 15 k' times in total, so that each distorted image in the image set belongs to the test set only once, and obtain the objective quality evaluation prediction value of each distorted image in the image set.
2. A method for evaluating the quality of panoramic stitching images according to claim 1, Features In the step 2, and and and The acquisition process is: Step 2_1: I S Convert from the equirectangular coordinate system to the geometric coordinate system to obtain the corresponding spherical panoramic image, denoted as I sphere,S , will I sphere,S The pixel value of the pixel point with the coordinate position (φ,θ) is recorded as I sphere,S (φ,θ), Among them, φ is the longitude coordinate in the geometric coordinate system, θ is the latitude coordinate in the geometric coordinate system, -π≤φ≤π, φ=0 and φ=π correspond to the positions of the two seams; Step 2_2: I sphere,S Along N 1 The coordinate projection of the spherical center point of the spliced viewports is N 1 Distorted stitched viewport images, I sphere,S The coordinates of the center point of the sphere along the i-th stitching viewport The i-th distorted spliced viewport image obtained by projection is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Among them, N 1 ≥1, 1≤i≤N 1 , Represents the coordinate position Right now Represents the longitude coordinate of the spherical center point of the i-th stitching viewport, Represents the latitude coordinate of the spherical center point of the i-th stitching viewport, Represents the longitude coordinate of the i-th stitching viewport at the spherical sampling point, Represents the latitude coordinate of the i-th stitching viewport at the spherical sampling point, The step size is The step size is α i Represents the i-th distorted stitched viewport image The viewing angle size; Step 2_3: Use panoramic convolution 360° single image saliency prediction method to extract I S The saliency map of A , will I A The pixel value of the pixel point with coordinate position (u,v) is I S The saliency value of the pixel with coordinate position (u, v) is recorded as I A (u,v); then for I A The coordinate positions of all pixels in are arranged in descending order according to the pixel values of the pixels, and the set of arranged coordinate positions is recorded as {P 1 ,P 2 ,P 3 ,...,P W×H }; Among them, P 1 =(u 1 ,v 1 ) represents the first coordinate position after arrangement, which is I A The coordinate position of the pixel with the largest pixel value, P 2 =(u 2 ,v 2 ) represents the second coordinate position after arrangement, which is I A The coordinate position of the pixel with the second largest pixel value, P 3 =(u 3 ,v 3 ) represents the third coordinate position after arrangement, which is I A The coordinate position of the pixel with the third largest pixel value, P W×H =(u W×H ,v W×H ) represents the W×Hth coordinate position after arrangement, which is I A The coordinate position of the pixel with the smallest pixel value; Step 2_4: Select the coordinates of the spherical center point of the first significant viewport The specific process is: Traverse {P 1 ,P 2 ,P 3 ,...,P W×H }, P 1 Convert from the equirectangular coordinate system to the geometric coordinate system to obtain the transformed coordinate position, denoted as P 1 ',P 1 '=(φ 1 ,θ 1 ), Then calculate P 1 'With N 1 The angular distance of the coordinates of the spherical center point of the stitching viewport, P 1 'Coordinates of the spherical center point of the i-th stitching viewport The angular distance is denoted as d i , Then get the minimum angle distance, recorded as d min , d min =min(d i |1≤i≤N 1 );Then judge d min Is it greater than the threshold d? th , if d min Greater than the threshold d th , then P 1 'As the coordinates of the spherical center point of the first significant viewport, re-recorded as If d min Not greater than the threshold d th , then traverse P 2 , press the 1 The process is the same as first obtaining P 2 The transformed coordinate position P 2 ', then calculate the transformed coordinate position P 2 'With N 1 The angular distance of the coordinates of the spherical center point of the stitched viewports, and then obtain the minimum angular distance. Finally, if the minimum angular distance is greater than the threshold d th , then P 2 'As the coordinates of the spherical center point of the first significant viewport, re-recorded as If the minimum angle distance is not greater than the threshold d th , then traverse P 3 , and so on, until the minimum angle distance is found to be greater than the threshold d th Under the condition of b , P b The transformed coordinate position P b 'As the coordinates of the spherical center point of the first significant viewport, re-recorded as Among them, min() is the minimum function, P b ∈{P 1 ,P 2 ,P 3 ,...,P W×H }, P b =(u b ,v b ) represents the bth coordinate position after arrangement, b∈[1,W×H], P 2 '=(φ 2 ,θ 2 ), P b '=(φ b ,θ b ), The longitude coordinate of the spherical center point of the first significant viewport, The latitude coordinate of the spherical center point of the first significant viewport; Step 2_5: Select the 2nd to Nth 2 The coordinates of the spherical center point of the prominent viewport, for the selection of The coordinates of the spherical center point of the prominent viewport The specific process is: Set the selected The coordinates of the spherical center point of the prominent viewport Corresponding to {P 1 ,P 2 ,P 3 ,...,P W×H } Then in {P 1 ,P 2 ,P 3 ,...,P W×H } Start traversing each subsequent coordinate position in turn, Convert from the equirectangular coordinate system to the geometric coordinate system to obtain the transformed coordinate position, recorded as Then calculate separately With N 1 The angular distance of the coordinates of the spherical center points of the stitched viewports, The coordinates of the spherical center point of the i-th stitching viewport The angular distance is recorded as Then get the minimum angle distance, recorded as And calculate separately With the selected 1st to The angular distance between the coordinates of the spherical center point of the significant viewport, With The angular distance between the coordinates of the spherical center points of the prominent viewports is recorded as Then get the minimum angle distance, recorded as Re-judge and Are they all greater than the threshold d? th ,if and Both are greater than the threshold d th , then As the The coordinates of the spherical center point of the prominent viewport are rewritten as If not satisfied and Both are greater than the threshold d th , then traverse According to the target The process is the same as first obtaining Transformed coordinate position Then calculate the transformed coordinate position With N 1 The angular distance of the coordinates of the spherical center points of the stitched viewports and the minimum angular distance are obtained, and the transformed coordinate position is calculated The angular distance from the coordinates of the spherical center points of all selected significant viewports and the minimum angular distance are obtained. Finally, if both minimum angular distances are greater than the threshold d th , then As the The coordinates of the spherical center point of the prominent viewport are rewritten as If the two minimum angle distances obtained are not greater than the threshold d th , then traverse And so on, until the two minimum angular distances are found to be greater than the threshold d th The coordinate position under the condition of 2 ≥1, The initial value of is 2. Indicates The longitude coordinates of the spherical center points of the prominent viewports, Indicates The latitude coordinates of the spherical center points of the prominent viewports, Indicates the first Coordinate positions, Indicates the first Coordinate positions, Step 2_6: The coordinates of the spherical center point of the first significant viewport obtained in step 2_4 and the 2nd to Nth obtained in step 2_5 2 The coordinates of the spherical center points of the prominent viewports form a set, denoted as Then I sphere,S Along N 2 The coordinate projection of the spherical center point of the salient viewports is N 2 A distorted viewport image, I sphere,S The coordinates of the sphere center point along the jth salient viewport The jth distorted viewport image obtained by projection is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Among them, 1≤j≤N 2 , represents the coordinates of the spherical center point of the jth salient viewport, represents the longitude coordinate of the spherical center point of the jth significant viewport, represents the latitude coordinate of the spherical center point of the jth salient viewport, represents the longitude coordinate of the jth significant viewport at the spherical sampling point, represents the latitude coordinate of the jth significant viewport at the spherical sampling point, The step size is The step size is β j represents the jth distorted viewport image The viewing angle size; Follow steps 2_1 to 2_6 to obtain I S The corresponding N 1 distorted stitched viewport images and N 2 In the same way, I G The corresponding N 1 reference stitched viewport images and N 2 A reference salient viewport image, I G The corresponding i-th reference stitching viewport image is recorded as Will I G The corresponding j-th reference salient viewport image is recorded as 3. A method for evaluating the quality of panoramic stitching images according to claim 1 or 2, Features In the step 5, The acquisition process is: Step 5_1: Convert to grayscale image, denoted as Then calculate The guiding matrix corresponding to each pixel in The steering matrix corresponding to the qth pixel in is recorded as in, express The total number of pixels in Ω q Indicates The coordinate position of the qth pixel in The set of coordinate positions of all pixels in the rectangular window with the center point as the center point, Indicates The coordinate position of the qth pixel in The coordinate position of the q'th pixel in the rectangular window with the center point as the center point, Indicates The coordinate position of the qth pixel in The total number of pixels in the rectangular window with the center point as the center point, express China-Israel The coordinate position of the qth pixel in is the pixel value of the q'th pixel in the rectangular window at the center point, Indicates The coordinate position of the qth pixel in The horizontal coordinate of the q'th pixel in the rectangular window with the center point as the center point, Indicates The coordinate position of the qth pixel in The ordinate of the q'th pixel in the rectangular window with the center point as the center point, symbol To find the sign of partial derivative; Step 5_2: Calculation The similarity between each pixel in the p×p neighborhood around the central pixel is The qth pixel in the image and the p×p neighbors around it as the center pixel The first The similarity between neighboring pixels is recorded as Then get The local guide kernel feature vector of each pixel in The local guide kernel feature vector of the qth pixel in is recorded as Among them, p ≥ 3, exp() represents an exponential function with the natural base e as the base. express China-Israel The qth pixel in the p×p neighborhood around the central pixel The coordinate position of the neighboring pixel points, if exist Outside the scope The dimension of h is 1×(p×p-1), q,1 express The similarity between the qth pixel in the image and the first neighboring pixel in the p×p neighborhood around the center pixel, h q,2 express The similarity between the qth pixel in the image and the second neighboring pixel in the p×p neighborhood around the centered pixel, h q,p×p-1 express The similarity between the qth pixel in and the p×p-1th neighboring pixel in the p×p neighborhood around the centered pixel; Step 5_3: Combination The local guide kernel feature vectors of all pixels in are obtained The local guide kernel feature tensor of Will The mid-coordinate position is The pixel value of the pixel point is recorded as Then principal component analysis was used to extract The first principal component, the second principal component, and the third principal component are recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Step 5_4: Follow the process from step 5_1 to step 5_3 to obtain The grayscale image is calculated by The guidance matrix corresponding to each pixel in the grayscale image is obtained The local guided kernel feature tensor of the grayscale image Re-extraction The first principal component, the second principal component, and the third principal component are recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Step 5_5: Calculation The mean pixel value of the ξth superpixel block in is denoted as Similarly, calculation The mean pixel value of the ξth superpixel block in is denoted as calculate The mean pixel value of the ξth superpixel block in is denoted as Calculated in Zhongyu The mean pixel value of the superpixel block matched by the ξth superpixel block in is denoted as Calculated in Zhongyu The mean pixel value of the superpixel block matched by the ξth superpixel block in is denoted as Calculated in Zhongyu The mean pixel value of the superpixel block matched by the ξth superpixel block in is denoted as in, Representation Statistics The total number of pixels in the ξth superpixel block in , Indicates that statistics Zhongyu The total number of pixels in the superpixel block matched by the ξ-th superpixel block in ; Step 5_6: Calculate Zhongyu The edge-aware features of the superpixel block matched by the ξth superpixel block in Calculated in Zhongyu The edge-aware features of the superpixel block matched by the ξth superpixel block in Calculated in Zhongyu The edge-aware features of the superpixel block matched by the ξth superpixel block in Step 5_7: Concatenate to form a feature vector, as 4. The method for evaluating the quality of panoramic stitching images according to claim 1, Features In step 6, The acquisition process is: Step 6_1: Use aspect ratio similarity evaluation method to obtain The ξth superpixel block in Zhongyu The aspect ratio similarity of the superpixel block matched by the ξth superpixel block in is denoted as according to and Calculated; Step 6_2: Extraction All boundary points in the ξ-th superpixel block in form a set, denoted as Similarly, extract Zhongyu All boundary points in the superpixel block matched by the ξth superpixel block in , and form a set, recorded as Then a dynamic time planning algorithm is used to obtain The ξth superpixel block in Zhongyu The boundary similarity of the superpixel block matched by the ξth superpixel block in is denoted as according to and Calculated; Step 6_3: and Concatenate to form a feature vector, as 5. The method for evaluating the quality of panoramic stitching images according to claim 1, Features In the step 7, The acquisition process is: Step 7_1: In the HSV color space, the H component, S component, and V component are recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Step 7_2: In the HSV color space, the H component, S component, and V component are recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Will The pixel value of the pixel point with coordinate position is recorded as Will The mid-coordinate position is The pixel value of the pixel point is recorded as Step 7_3: Calculation The mean value of the hue information of the ξth superpixel block in is denoted as And calculated in Zhongyu The mean value of the hue information of the superpixel block matched by the ξth superpixel block in Then calculate Zhongyu The color perception feature of the H channel of the superpixel block matched by the ξth superpixel block in is recorded as Similarly, calculation The mean saturation information of the ξth superpixel block in is denoted as And calculated in Zhongyu The mean saturation information of the superpixel block matched by the ξth superpixel block in is recorded as Then calculate Zhongyu The color perception feature of the S channel of the superpixel block matched by the ξth superpixel block in is recorded as Similarly, calculation The mean brightness information of the ξth superpixel block in is denoted as And calculated in Zhongyu The mean brightness information of the superpixel block matched by the ξth superpixel block in is recorded as Then calculate Zhongyu The color perception feature of the V channel of the superpixel block matched by the ξth superpixel block in is recorded as in, Representation Statistics The total number of pixels in the ξth superpixel block in , Indicates that statistics Zhongyu The total number of pixels in the superpixel block matched by the ξ-th superpixel block in ; Step 7_3: Concatenate to form a feature vector, as 6. A method for evaluating the quality of panoramic stitching images according to claim 3, Features In the step 8, The acquisition process is: Step 8_1: Calculation The structural complexity weight of each superpixel block in The structural complexity weight of the ξth superpixel block in is recorded as Among them, exp() represents the exponential function with the natural base e as the base, σ is the parameter used to control the saturation point of the nonlinear function, express The average edge strength of the ξth superpixel block in express The mean pixel value of the ξth superpixel block in express The mean pixel value of the ξth superpixel block in express The mean pixel value of the ξth superpixel block in express The first principal component is obtained by principal component analysis of the local guided kernel feature tensor of the grayscale image. express The second principal component is obtained by principal component analysis of the local guided kernel feature tensor of the grayscale image. express The third principal component is obtained by principal component analysis of the local guided kernel feature tensor of the grayscale image; Step 8_2: Calculation The matching confidence weight of each superpixel block in The matching confidence weight of the ξth superpixel block in is recorded as in, Representation Statistics The total number of pixels in the ξth superpixel block in , δ is a constant used to improve stability, Indicates based on and The pixel matching relationship between The reference viewport image reconstructed from the matching pixels in The mid-coordinate position is The pixel value of the pixel point; Step 8_3: Calculation The final weight of each superpixel block in The final weight of the ξ-th superpixel block in is recorded as Then get
