A Stereoscopic Panoramic Image Objective Quality Assessment System and Method

Abstract

A system and method for objectively evaluating the quality of a stereoscopic panoramic image, which relates to the technical field of no-reference quality assessment of stereoscopic panoramic images. It solves the problems that existing objective evaluation methods for stereoscopic panoramic images generally have weak universality, low accuracy in quality evaluation, and long time consumption and high cost. The method is as follows: according to the left and right views of the stereoscopic panoramic image, N left and right viewport images are obtained and processed to obtain the final left local features and the final right local features; the ERP images of the left and right views are processed to obtain the left view global features and the right view global features; the final left local features are linearly combined with the left view global features to obtain the left fusion features; the final right local features are linearly combined with the right view global features to obtain the right fusion features; and the two are processed to obtain the left-right mutual attention map, and the mutual attention map is mapped to obtain the quality evaluation score. The present invention is applicable to the quality assessment of stereoscopic panoramic images.

Description

Technical Field

[0001] The present invention relates to the technical field of no-reference quality assessment of stereoscopic panoramic images. Background Art

[0002] With the rapid development of VR technology and the widespread adoption of head-mounted displays (HMDs), people are now able to apply VR technology in various areas of daily life, including leisure and entertainment, commercial services, and healthcare. At the same time, people's demand for immersive and stereoscopic viewing is increasing, which is also prompting VR content creators to improve their technology and provide users with a better experience. To ensure optimal subjective perception of quality, it is necessary to quantify the degree of image quality degradation so that we can optimize the image in subsequent processing and achieve image quality control.

[0003] In most cases, the human eye is the final receiver of an image, making subjective evaluation methods more consistent with human perception. However, subjective evaluation is susceptible to individual factors such as subject age, number, psychological state, and experience. While the influence of individual factors can be mitigated by increasing the number of test subjects, controlling experimental conditions, and eliminating outliers, current subjective evaluation methods are still time-consuming and costly, and there is a lack of effective harmonization between different methods. Therefore, developing objective evaluation methods that are consistent with subjective evaluation is of great significance.

[0004] Objective quality evaluation of stereoscopic panoramic images requires a clear understanding of their characteristics and targeted design. Existing objective evaluation methods for stereoscopic panoramic images mostly design models based on the characteristics of stereoscopic and panoramic images, perform manual feature extraction, and fuse them together, or simply modify 2D image quality evaluation methods and apply them to stereoscopic panoramic images. However, stereoscopic panoramic images are not simply a combination of stereoscopic and panoramic images, and manually designing and fusing features based on the characteristics of both does not fully reflect the characteristics of stereoscopic panoramic images. Therefore, existing objective evaluation methods for stereoscopic panoramic images generally suffer from weak universality and low quality evaluation accuracy. Summary of the Invention

[0005] The present invention solves the problems of existing objective evaluation methods for stereoscopic panoramic images, such as weak universality, low quality evaluation accuracy, long time consumption and high cost.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] The present invention provides a method for evaluating the quality of a stereoscopic panoramic image without reference, wherein the method comprises:

[0008] S1, extracting the left view and the right view of the stereo panoramic image;

[0009] S2. Obtain N left viewport images according to the left view of the stereoscopic panoramic image;

[0010] S3. Obtain N right viewport images according to the right view of the stereoscopic panoramic image;

[0011] S4. Process the left viewport images to obtain N left local adaptive features;

[0012] S5. Process the right viewport images to obtain N right local adaptive features;

[0013] S6. Process the ERP image of the left view using the Swin Transformer model to obtain the global feature of the left view;

[0014] S7. Process the ERP image of the right view using the Swin Transformer model to obtain the global feature of the right view;

[0015] S8. Use the averaging method to fuse the N left local adaptive features to obtain the final left local feature;

[0016] S9. Use the averaging method to fuse the N right local adaptive features to obtain the final right local feature;

[0017] S10. Use a linear layer to linearly combine the final left local feature and the global feature of the left view to obtain the left fusion feature;

[0018] S11. Use a linear layer to linearly combine the final right local feature and the global feature of the right view to obtain the right fusion feature;

[0019] S12. Perform interactive processing on the left fusion feature and the right fusion feature to obtain the left-right mutual attention map;

[0020] S13. Use a fully connected layer to map the mutual attention map to obtain the quality evaluation score.

[0021] Furthermore, there is also a preferred embodiment. Specifically, step S2 is as follows:

[0022] S21. Use SURF features to select the left view of the panoramic image to obtain the left view key point distribution map;

[0023] S22. Perform convolution processing on the left view key point distribution map to obtain the left view heat map;

[0024] S23. Process the left view heat map to obtain N left viewport images.

[0025] Furthermore, there is also a preferred embodiment. In the above step S21, the left view key point distribution map is represented as:

[0026] point_map = detectSURFFeatures(erp);

[0027] where erp represents the input left / right view image, and detectSURFFeatures is the key point selection algorithm.

[0028] Furthermore, there is also a preferred embodiment. The above step S4 is specifically:

[0029] S41. Use a convolutional neural network to extract the left viewport image to obtain the local features of the left viewport image;

[0030] S42. Use a convolutional operation to extract the local features of the left viewport image to obtain the offset of the left local features;

[0031] S43. Use a deformable convolutional operation to extract the offset of the left local features and the local features of the left viewport image to obtain the adaptive features of N left local features.

[0032] Furthermore, there is also a preferred embodiment. In the above step S8, the final left local features are represented as:

[0033]

[0034] where K is the local feature index of the viewport image, viewport is the viewport image, and F deform is the adaptive feature of the viewport image.

[0035] Furthermore, there is also a preferred embodiment. In the above step S10, the left fusion features are represented as:

[0036] F L = Linear(cat(F fusion , F global ));

[0037] where F global is the global feature, cat is to concatenate F fusion and F global , and Linear represents the linear layer.

[0038] Furthermore, there is also a preferred embodiment. In the above step S12, the left-right mutual attention map is represented as:

[0039]

[0040] where F L and FR represent the left - view and right - view features respectively, d represents the dimension of the feature, T is the matrix transpose, and Softmax() is used to generate self - attention weights.

[0041] The method for non - reference quality assessment of stereoscopic panoramic images described in the present invention can be fully implemented by computer software. Therefore, correspondingly, the present invention also provides a non - reference quality assessment system for stereoscopic panoramic images, and the system includes:

[0042] A storage device for extracting the left view and right view of the stereoscopic panoramic image;

[0043] A storage device for obtaining N left - viewport images according to the left view of the stereoscopic panoramic image;

[0044] A storage device for obtaining N right - viewport images according to the right view of the stereoscopic panoramic image;

[0045] A storage device for processing the left - viewport images to obtain N left local adaptive features;

[0046] A storage device for processing the right - viewport images to obtain N right local adaptive features;

[0047] A storage device for processing the ERP image of the left view by using the Swin Transformer model to obtain the global feature of the left view;

[0048] A storage device for processing the ERP image of the right view by using the Swin Transformer model to obtain the global feature of the right view;

[0049] A storage device for fusing the N left local adaptive features by using the averaging method to obtain the final left local feature;

[0050] A storage device for fusing the N right local adaptive features by using the averaging method to obtain the final right local feature;

[0051] A storage device for linearly combining the final left local feature and the global feature of the left view by using a linear layer to obtain the left fusion feature;

[0052] A storage device for linearly combining the final right local feature and the global feature of the right view by using a linear layer to obtain the right fusion feature;

[0053] A storage device for performing interactive processing on the left fusion feature and the right fusion feature to obtain the left - right mutual attention map;

[0054] A storage device for mapping the mutual attention map using a fully connected layer to obtain a quality evaluation score.

[0055] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, it executes a method for non-reference quality assessment of a stereoscopic panoramic image described in any one of the above.

[0056] The present invention also provides a computer device, which includes a memory and a processor. A computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes a method for non-reference quality assessment of a stereoscopic panoramic image described in any one of the above.

[0057] The beneficial effects of the present invention are as follows:

[0058] 1. The present invention provides a method for non-reference quality assessment of a stereoscopic panoramic image, which uses a global feature plus local feature extraction method to simulate the visual perception of humans when viewing a panoramic image. The method represents the observation area of the head-mounted display when humans view VR content by extracting the viewport image, and combines the complete ERP image to represent the overall perception of the complete VR content. Compared with the prior art that directly extracts features only in the ERP format, this method is more in line with the process of humans viewing VR content.

[0059] 2. The present invention provides a method for non-reference quality assessment of a stereoscopic panoramic image, which uses a convolutional operation to obtain the offset of local features, so that the shape and position of the convolutional kernel of the subsequent deformable convolution can be dynamically adjusted according to the content of the current feature, thereby adapting to different object sizes and geometric shapes, and uses deformable convolution to extract adaptive features of different images. Traditional convolutional neural networks will mix other background information unrelated to the object when extracting object features, while the deformable convolution described in the present invention can improve the flexibility of the receptive field shape. It can adaptively select the region of interest according to the shape of the object to simulate the orientation selectivity reflected by the stimulation of the primary visual cortex, and at the same time enhance the network's ability to capture uneven multi-scale features.

[0060] 3. The present invention provides a method for non-reference quality assessment of a stereoscopic panoramic image, which uses the Swin Transformer to focus on extracting global features from the overall image, supplementing the lack of global modeling ability in the application of CNN under the viewport map. At the same time, it organically combines local features and global features to enhance the feature expression ability of the model.

[0061] 4. The present invention provides a method for evaluating the quality of stereoscopic panoramic images without reference, which uses cross-attention for left and right view interaction, simulates the fusion and competition phenomena of the left and right eyes when humans view stereoscopic images, generates left and right interactive attention maps based on the left and right views, and promotes the cross-view interaction ability of the model.

[0062] The present invention is applicable to the quality evaluation of stereoscopic panoramic images. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 is a schematic flowchart of a method for evaluating the quality of stereoscopic panoramic images without reference according to Embodiment 1;

[0064] Figure 2 is the extracted stereoscopic panoramic viewport image according to Embodiment 1;

[0065] Figure 3 is the left view of the segmented stereoscopic panoramic image according to Embodiment 1. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0066] The following further elaborates on the specific embodiments of the present invention in conjunction with the drawings and examples. The following examples will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several changes and improvements can still be made, and these all fall within the protection scope of the present invention.

[0067] Embodiment 1. Refer to Figures 1 to 3 To illustrate this embodiment, this embodiment provides a method for evaluating the quality of stereoscopic panoramic images without reference, and the evaluation method is as follows:

[0068] S1. Extract the left view and right view of the stereoscopic panoramic image;

[0069] S2. Obtain N left viewport images according to the left view of the stereoscopic panoramic image;

[0070] S3. Obtain N right viewport images according to the right view of the stereoscopic panoramic image;

[0071] S4. Process the left viewport images to obtain N left local adaptive features;

[0072] S5. Process the right viewport images to obtain N right local adaptive features;

[0073] S6. Process the ERP image of the left view using the Swin Transformer model to obtain the global features of the left view;

[0074] S7. Process the ERP image of the right view using the Swin Transformer model to obtain the global features of the right view;

[0075] S8. Use the method of taking the average to fuse the N left local adaptive features to obtain the final left local features;

[0076] S9. Use the method of taking the average to fuse the N right local adaptive features to obtain the final right local features;

[0077] S10. Use a linear layer to linearly combine the final left local features and the global features of the left view to obtain the left fusion features;

[0078] S11. Use a linear layer to linearly combine the final right local features and the global features of the right view to obtain the right fusion features;

[0079] S12. Perform interactive processing on the left fusion features and the right fusion features to obtain the left-right mutual attention map;

[0080] S13. Use a fully connected layer to map the mutual attention map to obtain the quality evaluation score.

[0081] In actual application of this embodiment, the complete image in the stereo panoramic image dataset is spliced in the up-down manner for the left and right views. First, separate the left and right views, and extract the left view and the right view of the stereo panoramic image respectively as the input of the model. Here, erp_l represents the left view of the stereo panoramic image, and erp_r represents the right view of the stereo panoramic image, as Figure 3 shown. According to the left view of the stereo panoramic image, obtain the left viewport image, and according to the right view of the stereo panoramic image, obtain the right viewport image, as Figure 2 shown. Process the left viewport image to obtain N left local adaptive features; process the right viewport image to obtain N right local adaptive features; then use the method of taking the average to fuse the N left local adaptive features respectively to obtain the final left local features; fuse the N right local adaptive features to obtain the final right local features; use the Swin Transformer model to process the ERP image of the left view to obtain the global features of the left view, and use the Swin Transformer model to process the ERP image of the right view to obtain the global features of the right view. Specifically: input the ERP images erp of the left and right views into the Swin Transformer architecture respectively, which includes image processing and four stage modules. The erp passes through image processing to obtain F embedding, it is input into the subsequent stage, and each stage contains multiple Swin Transformer Blocks and a Patch Merging. The Swin Transformer Block contains Window Attention and Shifted-Window Attention. The former belongs to the self-attention of the sliding window, and the latter belongs to the self-attention of the shifted sliding window.

[0082] Self-attention is used to explore the importance of each feature vector, paying more attention to useful information and ignoring useless information. First, calculate the query (Q), key (K), and value (V) of self-attention:

[0083] Q = W q *F embedding ;

[0084] K = W k *F embedding ;

[0085] V = W v *F embedding ;

[0086] Wq, Wk, and Wv represent the weight matrices respectively, and Q, K, and V represent the calculated query, key, and value respectively. Calculate the self-attention of the features according to the obtained Q, K, and V:

[0087]

[0088] d represents the dimension of the feature, Softmax() is used to generate the self-attention weight, T represents the matrix transpose, B represents the added relative position encoding, which is used to distinguish the relative position information of each F embedding of, F self-attention represents the feature after the interaction of F embedding within the window (window). In order to interact with other windows, the Shifted-Window operation is introduced. The spatial feature values with a width of window / 2 on the left and above are respectively moved to the right and below, and the upper left part is moved to the lower right corner. At the same time, in order to ensure that non-adjacent boundary features do not perform self-attention, a masking operation is added to the feature map before softmax, making the non-close features negative infinity after masking, and the weight will be set to 0 during self-attention calculation. The formula is as follows.

[0089]

[0090] where F shifted_self-attentionIt represents the self-attention calculation after the sliding window, and mask represents the masking operation.

[0091] The structural formula of each Swin Transformer module is as follows.

[0092] F window = F embedding + MLP(LN(F self-attention (LN(F embedding ))));

[0093] F swin = F window + MLP(LN(F shifted_self-attention (LN(F window ))));

[0094] Where LN represents the normalization operation, MLP represents the multi-layer perceptron, and F window represents the feature obtained after window attention, and F swin represents the feature after passing through the Swin Transformer module.

[0095] The formula for each stage is as follows.

[0096] F stage = F patch_merging (F swin );

[0097] Where F patch_merging () represents the downsampling operation, which is used to reduce the resolution, adjust the number of channels, and thus form a hierarchical design while reducing the complexity. F stage represents the feature after passing through each stage structure. The overall formula is as follows:

[0098]

[0099] Where F global represents the global feature extracted after passing through the overall Swin Transformer, and j represents the stage module index.

[0100] Then, a linear layer is used to linearly combine the final left local feature and the global feature of the left view to obtain the left fusion feature. A linear layer is used to linearly combine the final right local feature and the global feature of the right view to obtain the right fusion feature. The left fusion feature and the right fusion feature are interactively processed to obtain the left-right mutual attention map. A fully connected layer is used to map the mutual attention map to obtain the quality evaluation score. Specifically: The final feature is mapped into a predicted objective score through a fully connected layer. The formula is as follows:

[0101] Score = FC(Fcross );

[0102] FC represents the fully connected layer, Score represents the predicted objective score, and F cross represents the cross-view interaction feature.

[0103] To solve the problems existing in the existing objective evaluation methods for stereoscopic panoramic images, such as relatively weak universality, low accuracy of quality evaluation, long time consumption, and high cost.

[0104] This embodiment provides a method for non-reference quality assessment of stereoscopic panoramic images, which uses the global feature plus local feature extraction method to simulate the visual perception of humans when viewing panoramic images. The way of extracting the viewport image is used to represent the observation area of the head-mounted display when humans view VR content, and the complete ERP image is combined to represent the overall perception of the complete VR content. Compared with the prior art that directly extracts features only in the ERP format, this method is more in line with the process of humans viewing VR content.

[0105] This embodiment provides a method for non-reference quality assessment of stereoscopic panoramic images, which uses convolution operations to obtain the offset of local features, so that the convolution kernel shape and position of subsequent deformable convolution can be dynamically adjusted according to the content of the current features, thereby adapting to different object sizes and geometric shapes, and uses deformable convolution to extract the adaptive features of different images. Traditional convolutional neural networks will mix other background information unrelated to the object when extracting object features, while the deformable convolution described in the present invention can improve the flexibility of the receptive field shape. It can adaptively select the area of interest according to the shape of the object to simulate the orientation selectivity reflected by the stimulation of the primary visual cortex, and at the same time enhance the ability of the network to capture uneven multi-scale features.

[0106] This embodiment provides a method for non-reference quality assessment of stereoscopic panoramic images, which uses the Swin Transformer to focus on extracting global features from the overall image, supplementing the lack of global modeling ability in the application of CNN under the viewport image. At the same time, it organically fuses local features and global features, enhancing the feature expression ability of the model.

[0107] This embodiment provides a method for non-reference quality assessment of stereoscopic panoramic images, which uses cross-attention for left and right view interaction, simulating the fusion and competition phenomena of the left and right eyes when humans view stereoscopic images, generating left and right interaction attention maps based on the left and right views, and promoting the cross-view interaction ability of the model.

[0108] Embodiment 2. This embodiment gives an example of step S2 in a method for non-reference quality assessment of stereoscopic panoramic images described in Embodiment 1, and the specific step S2 is as follows:

[0109] S21. Select the left view of the panoramic image using SURF features to obtain the left view key point distribution map;

[0110] S22. Perform convolution processing on the left view key point distribution map to obtain the left view heat map;

[0111] S23. Process the left view heat map to obtain N left viewport images.

[0112] In actual application of this embodiment, SURF features are used to select the key points of the panoramic image in the ERP format. Key points are points in the two-dimensional image that have color and gray-scale differences from the surrounding pixel points. According to the selected key points, a key point distribution map point_map is generated. The generation of key points is as follows:

[0113] point_map = detectSURFFeatures(erp);

[0114] Where erp represents the input left view image, detectSURFFeatures represents the key point selection algorithm, and point_map represents the key point distribution map generated by the selected key points.

[0115] The generated key point distribution map point_map is convolved to generate the heat map Heatmap. The maximum point is selected on the heat map. The angular distance between points is set to be greater than 30 degrees. The largest twenty are the center points of 20 viewports. The area where the center points spread to the surrounding to form 256 * 256 is the extracted left viewport image viewport. In application, the right view of the stereoscopic panoramic image can be operated according to the above steps to obtain the right viewport image.

[0116] Embodiment 3. This embodiment gives an example of the left view key point distribution map in step S21 of a method for non-reference quality assessment of a stereoscopic panoramic image described in Embodiment 2. The key point distribution map is expressed as:

[0117] point_map = detectSURFFeatures(erp);

[0118] Where erp represents the input left / right view image, and detectSURFFeatures is the key point selection algorithm.

[0119] In actual application of this embodiment, to obtain the key point distribution map of the left view, the left view image is input, and to obtain the key point distribution map of the right view, the right view image is input.

[0120] Embodiment 4. This embodiment gives an example of step S4 in a method for non-reference quality assessment of stereoscopic panoramic images described in Embodiment 1. The specific steps of step S4 are as follows:

[0121] S41. Use a convolutional neural network to extract the left viewport image to obtain the local features of the left viewport image.

[0122] S42. Use a convolutional operation to extract the local features of the left viewport image to obtain the offset of the left local features.

[0123] S43. Use a deformable convolutional operation to extract the offset of the left local features and the local features of the left viewport image to obtain the adaptive features of N left local features.

[0124] In actual application of this embodiment, the local features of the viewport image are extracted according to the following formula:

[0125]

[0126] Where F local represents the local features of the viewport image, ConvBlock represents the convolutional neural network module, i represents the index of the convolutional neural network module, and viewport represents the input left / right viewport image. The local features of the viewport image are extracted by the convolutional neural network for subsequent feature integration. The formula of the convolutional neural network module ConvBlock is as follows:

[0127] ConvBlock = Pool(CNN(viewport)) + Residual(viewport);

[0128] Where CNN represents the convolutional operation, Pool represents the pooling operation, and Residual represents the residual connection. The deep neural network CNN is used to extract features from the input viewport image, and the pooling layer Pool compresses the extracted feature map to reduce the spatial size of the feature map, simplify the network calculation complexity, and retain the main features of the image. Residual can well avoid the problem of gradient disappearance and is beneficial to the training of the model.

[0129] Calculate the offset of the local features so that the shape and position of the convolutional kernel of the subsequent deformable convolution can be dynamically adjusted according to the content of the current features, so as to adapt to different object sizes and geometric shapes. The offset calculation is as follows:

[0130] offsets = CNN(F local );

[0131] CNN represents the convolutional operation, and offsets represents the offset of the local features.

[0132] Use deformable convolution to extract adaptive features of different images. According to the above offsets, the convolution kernel can find the objects in the image and adaptively extract features according to the size and shape of the objects, and find the regions of interest in the image. The formula is as follows:

[0133] F deform =Deformable(F local ,offsets)

[0134] Deformable represents the deformable convolution operation, and F deform represents the adaptive features of the extracted viewport image. In actual applications, if the adaptive features of the left local features are required, the left viewport image is input, and if the adaptive features of the right local features are required, the right viewport image is input.

[0135] Embodiment 5. This embodiment is an example of the final left local features in step S8 of a stereoscopic panoramic image non-reference quality assessment method described in Embodiment 1. The final left local features are represented as:

[0136]

[0137] where K is the local feature index of the viewport image, viewport is the viewport image, and F deform is the adaptive feature of the viewport image.

[0138] In actual applications of this embodiment, the value of N is 20, and the local features of 20 viewports of the left view or the right view are fused into a whole by using the averaging method to obtain the local features of the viewport image of the final left view or the local features of the viewport image of the final right view for subsequent local and global feature fusion.

[0139]

[0140] where F fusion represents the local features after fusing the viewports, and k represents the local feature index of the viewport image.

[0141] In applications, if the local features of the viewport image of the final left view are required, the left local adaptive features are input, and if the final right local features are required, the right local adaptive features are input for fusion.

[0142] Embodiment 6. This embodiment is an example of the left fusion features in step S10 of a stereoscopic panoramic image non-reference quality assessment method described in Embodiment 1. The left fusion features are represented as:

[0143] F L =Linear(cat(F fusion ,Fglobal ));

[0144] Among them, F global is the global feature, cat is to splice F fusion and F global together, and Linear represents a linear layer.

[0145] In actual application of this embodiment, the linear layer is used to linearly combine the local feature and the global feature.

[0146] F L / = Linear(cat(F fusion , F global ))

[0147] cat represents splicing F fusion and F global together, Linear represents a linear layer, and F L / respectively represent the local and global fusion features of the left / right views.

[0148] In application, if the left fusion feature needs to be obtained, the final left local feature and the left view global feature are linearly combined. If the right fusion feature needs to be obtained, the final right local feature and the right view global feature are linearly combined.

[0149] Embodiment Seven. This embodiment gives an example of the left and right mutual attention maps in step S12 of the method for non-reference quality assessment of a stereoscopic panoramic image described in Embodiment One. The left and right mutual attention maps are expressed as:

[0150]

[0151] Among them, F L and F R respectively represent the left view and right view features, d represents the dimension of the feature, T is the matrix transpose, and Softmax() is used to generate self-attention weights.

[0152] In actual application of this embodiment, after obtaining the left fusion feature and the right fusion feature, the left and right views will respectively generate an F L and an F R feature. When the human eye views a stereoscopic image, the images seen by the left and right eyes will produce a fusion and competition phenomenon. We use cross-attention to simulate this phenomenon. Cross-attention uses the left and right views as each other's key information to generate a mutual attention map to promote cross-view feature fusion. The formula is as follows, generating a mutual attention map based on the left and right view features:

[0153]

[0154] Among which, F cross represents the cross-view interaction feature between the left view and the right view. F L and F R respectively represent the left view and right view features, and d represents the dimension of the feature.

[0155] Embodiment 8. This embodiment provides a stereoscopic panoramic image non-reference quality assessment system, and the assessment system is as follows:

[0156] A storage device for extracting the left view and right view of the stereoscopic panoramic image;

[0157] A storage device for obtaining N left viewport images according to the left view of the stereoscopic panoramic image;

[0158] A storage device for obtaining N right viewport images according to the right view of the stereoscopic panoramic image;

[0159] A storage device for processing the left viewport images to obtain N left local adaptive features;

[0160] A storage device for processing the right viewport images to obtain N right local adaptive features;

[0161] A storage device for processing the ERP image of the left view by using the Swin Transformer model to obtain the global feature of the left view;

[0162] A storage device for processing the ERP image of the right view by using the Swin Transformer model to obtain the global feature of the right view;

[0163] A storage device for fusing the N left local adaptive features by using the averaging method to obtain the final left local feature;

[0164] A storage device for fusing the N right local adaptive features by using the averaging method to obtain the final right local feature;

[0165] A storage device for linearly combining the final left local feature and the global feature of the left view by using a linear layer to obtain the left fusion feature;

[0166] A storage device for linearly combining the final right local feature and the global feature of the right view by using a linear layer to obtain the right fusion feature;

[0167] A storage device for performing interaction processing on the left fusion feature and the right fusion feature to obtain the left-right mutual attention map;

[0168] A storage device for mapping the mutual attention map using a fully connected layer to obtain a quality evaluation score.

[0169] Embodiment Nine. This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, it executes a method for non-reference quality assessment of a stereoscopic panoramic image according to any one of Embodiments One to Seven.

[0170] Embodiment Ten. This embodiment provides a computer device, which includes a memory and a processor. A computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes a method for non-reference quality assessment of a stereoscopic panoramic image according to any one of Embodiments One to Seven.

[0171] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0172] In the description of this specification, the description referring to terms such as "an embodiment", "some embodiments", "an example", "a specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0173] The above are only embodiments of the present invention and do not limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the scope of the claims of the present invention.

Claims

1. A no-reference quality assessment method for stereo panoramic images, characterized in that: The evaluation method is: S1, extracting the left view and the right view of the stereo panoramic image; S2. Obtain N left viewport images according to the left view of the stereoscopic panoramic image; S3. Obtain N right viewport images according to the right view of the stereoscopic panoramic image; S4, processing the left viewport image to obtain N left local adaptive features; S5. Process the right viewport image to obtain N right local adaptive features; S6. Use the Swin Transformer model to process the ERP image of the left view to obtain the global features of the left view; S7, using the Swin Transformer model to process the ERP image of the right view to obtain the global features of the right view; S8. Fusing the N left local adaptive features by averaging to obtain a final left local feature; S9, fusing the N right local adaptive features by averaging to obtain a final right local feature; S10, using a linear layer to linearly combine the final left local feature and the left view global feature to obtain a left fused feature; S11, using a linear layer to linearly combine the final right local feature and the right view global feature to obtain a right fused feature; S12, interactively processing the left fusion feature and the right fusion feature to obtain a left-right mutual attention map; S13. Use a fully connected layer to map the mutual attention map to obtain a quality evaluation score.

2. The method for no-reference quality assessment of stereoscopic panoramic images according to claim 1, wherein: The step S2 is specifically as follows: S21, selecting the left view of the panoramic image using SURF features to obtain a key point distribution map of the left view; S22, performing convolution processing on the left view key point distribution map to obtain a left view heat map; S23: Process the left view heat map to obtain N left viewport images.

3. The method for no-reference quality assessment of stereoscopic panoramic images according to claim 2, wherein: The key point distribution diagram of the left view in step S21 is represented as follows: point_map=detectSURFFeatures(erp); Among them, erp represents the input left / right view image, and detectSURFFeatures is the key point selection algorithm.

4. The method for no-reference quality assessment of stereoscopic panoramic images according to claim 1, wherein: The step S4 is specifically as follows: S41, extracting the left viewport image using a convolutional neural network to obtain local features of the left viewport image; S42, extracting local features of the left viewport image using a convolution operation to obtain an offset of the left local feature; S43 , extracting the offset of the left local feature and the local feature of the left viewport image using a variable convolution operation to obtain adaptive features of N left local features.

5. The method for no-reference quality assessment of stereoscopic panoramic images according to claim 1, wherein: The final left local feature representation in step S8 is: Among them, K is the local feature index of the viewport image, viewport is the viewport image, F deform is the adaptive feature of the viewport image.

6. The method for no-reference quality assessment of stereoscopic panoramic images according to claim 1, wherein: The left fusion feature in step S10 is expressed as: F L =Linear(cat(F fusion ,F global )); Among them, F global is a global feature, cat is the F fusion With F global Put together, Linear represents the linear layer.

7. The method for no-reference quality assessment of stereoscopic panoramic images according to claim 1, wherein: The left and right mutual attention map in step S12 is expressed as: Among them, F L and F R Represent the left view and right view features respectively, d represents the dimension of the feature, T is the matrix transpose, and Softmax() is used to generate self-attention weights.

8. A no-reference quality assessment system for stereoscopic panoramic images, characterized in that: The evaluation system is: A storage device for extracting a left view and a right view of a stereoscopic panoramic image; a storage device for obtaining N left viewport images according to the left view of the stereoscopic panoramic image; a storage device for obtaining N right viewport images according to the right view of the stereoscopic panoramic image; a storage device for processing the left viewport image to obtain N left local adaptive features; a storage device for processing the right viewport image to obtain N right local adaptive features; A storage device for processing the ERP image of the left view using a Swin Transformer model to obtain global features of the left view; A storage device for processing the ERP image of the right view using a Swin Transformer model to obtain global features of the right view; A storage device for fusing the N left local adaptive features by an averaging method to obtain a final left local feature; A storage device for fusing the N right local adaptive features by an averaging method to obtain a final right local feature; A storage device for obtaining a left fused feature by linearly combining the final left local feature with the left view global feature using a linear layer; a storage device for linearly combining the final right local feature with the right view global feature using a linear layer to obtain a right fused feature; A storage device for interactively processing the left fusion feature and the right fusion feature to obtain a left-right mutual attention map; A storage device for mapping the mutual attention map using a fully connected layer to obtain a quality evaluation score.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the method for no-reference quality assessment of stereoscopic panoramic images according to any one of claims 1 to 7.

10. A computer device, characterized in that: The device includes a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the method for no-reference quality assessment of stereoscopic panoramic images according to any one of claims 1 to 7.

Images