Lightweight panoramic video quality evaluation method based on space-time redundancy
Through a lightweight panoramic video quality evaluation method based on space-time redundancy, the redundant information of 360° panoramic video is removed and the Video ViT model is constructed, which solves the problems of large computing volume and high resource consumption in the existing technology, and achieves faster and more Accurate 360° panoramic video quality evaluation, suitable for practical applications.
Patent Information
- Application Number
- CN202411005765.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-15
AI Technical Summary
The existing 360° panoramic video quality evaluation model has huge calculations and high resource consumption, which limits its widespread promotion in actual applications, and is not effective enough to evaluate compressed video quality, resulting in problems such as "virtual reality disease".
Through a lightweight panoramic video quality evaluation method based on space-time redundancy, 360° panoramic video is collected and preprocessed. The redundant information is removed by using isometric rectangular projection and space-time uniform sampling, the Video ViT model is constructed, the redundant features are deleted, and the monotony is calculated. to optimize the model and improve the evaluation efficiency.
Without losing accuracy, the calculation amount and resource usage are significantly reduced, the efficiency of 360° panoramic video quality evaluation is improved, suitable for actual scenarios, and higher accuracy and lower computing resource costs are obtained.
Smart Images

Figure CN118968078A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision, and in particular to a lightweight panoramic video quality evaluation method based on spatiotemporal redundancy. Background Art
[0002] As a new type of immersive technology, virtual reality (VR) is gradually emerging in many fields such as the automotive industry, healthcare, and entertainment. As an indispensable VR content, 360° panoramic video has penetrated into people's daily lives. Different from traditional 2D videos, 360° panoramic videos can provide users with a unique visual experience. They can change their field of view within a range of 360x180° to get an immersive experience. In order to meet the user's quality of experience (QoE) within the 360x180° viewing range, the resolution of 360° panoramic videos is usually very high, such as 8K, 16K or even higher, which will bring a huge burden to the storage and transmission of 360 videos. Therefore, in order to reduce this burden, it is inevitable to compress 360° panoramic videos to reduce the bit rate and bandwidth occupation. However, large-scale compression of 360° panoramic videos may cause "virtual reality diseases" such as dizziness, disorientation, and nausea, which seriously reduces the subjective QoE of viewers. In order to evaluate the compression performance, 360° panoramic video quality assessment is urgently needed to assess the quality degradation caused by compression and further guide VR system optimization to achieve better QoE.
[0003] In recent years, with the rapid development of 360° panoramic video technology and the popularization of high-definition and ultra-high-definition videos, the field of 360° panoramic video quality evaluation has also made significant progress. However, existing 360° panoramic video quality evaluation models often face the challenges of huge computational complexity and high resource consumption, which greatly limits their widespread promotion in practical applications. In response to the above problems, the present invention proposes a method for reducing the amount of model calculation by excluding redundant video information to achieve faster and more accurate 360° panoramic video quality evaluation. By deleting redundant spatiotemporal information in the 360° panoramic video, the quality characteristics of the 360° panoramic video are retained to the maximum extent, so that the efficiency of 360° panoramic video quality evaluation can be improved without losing accuracy. Summary of the invention
[0004] The present invention proposes a lightweight panoramic video quality evaluation method based on spatiotemporal redundancy, which can improve the efficiency of 360° panoramic video quality evaluation without losing accuracy as much as possible.
[0005] The present invention adopts the following technical solutions.
[0006] A lightweight panoramic video quality evaluation method based on spatiotemporal redundancy uses a panoramic video quality evaluation model and includes the following steps:
[0007] Step S1: Collect 360° panoramic videos, organize the videos into data sets and perform preprocessing;
[0008] Step S2: Map the dataset input into an equirectangular projection format and obtain the final model input after uniform sampling with spatiotemporal consistency;
[0009] Step S3: Remove redundant spatiotemporal information from the 360° panoramic video, use the retained features that highly represent the video quality, compare the predicted video quality score obtained by the model with the actual video quality score, and calculate their monotonicity loss, linear fusion loss, and mean square error to further optimize the model;
[0010] Step S4: input the 360° panoramic video to be detected into the panoramic video quality evaluation model, and judge the quality of the 360° panoramic video to be detected by the score predicted by the model.
[0011] In step S1, a 360° panoramic video in a correct video format required for evaluation is captured by a 360° panoramic video capture camera;
[0012] In step S4, the panoramic video quality assessment model is constructed according to the original architecture of the Video ViT model.
[0013] The step S1 is specifically as follows:
[0014] Step S11: Installing video acquisition equipment at the shooting site;
[0015] Step S12: Shoot a 360° panoramic original video and invite subjects to rate the shot 360° panoramic original video. After excluding outliers, take the average value as the true quality score of the current video, called y gt ;
[0016] Step S13: preprocess all 360° panoramic videos in the data set.
[0017] In step S13, the data preprocessing method includes adjusting the image size, image standardization and 360° panoramic original video true score standardization.
[0018] The step S2 is specifically as follows:
[0019] Step S21: The pre-processed 360° panoramic videos of different projection modes are uniformly mapped into an equidistant rectangular projection format, and the obtained video is called Video ERP ;
[0020] Step S22: Perform temporal and spatial uniform sampling on the pre-processed 360° panoramic video. The calculation method is as follows:
[0021]
[0022] Among them, N h and N w Respectively represent the number of samples in the video height and width, H and W represent Video ERp The height and width of rand_crop means to perform a random crop in a given area. h Indicates the height of vi h Positions, vj w Indicates the width of the vj w Locations, Video sample Indicates Video ERP The results obtained after uniform sampling in time and space;
[0023] Step S23: Video sample Input the 360° panoramic video quality assessment model for further processing.
[0024] In step S3, redundant spatiotemporal information in the 360° panoramic video is removed by simply encoding the model input.
[0025] The step S3 is specifically as follows:
[0026] Step S31: Video sample The input model is simply encoded through convolution, Feature encode =Conv(Video sample ), where Conv represents convolution calculation, Feature encode Indicates the result obtained after encoding;
[0027] Step S32: Feature encode The features obtained by the same rand_crop are deleted, and only one feature is retained at each position to represent the 360° panoramic video quality at that position. The result is called Fea t ure input ;
[0028] Step S33: Feature input The first, second, and third stage feature maps extracted by the self-attention layer are merged into a set of video quality feature maps of the dataset, called Feature extract ;
[0029] Step S34: Feature extract The first, second, and third stage video feature maps are expanded through the linear mapping layer to make Feature extractThe features of the first, second, and third stages have the same number of channels, and the expanded video feature map set is called Feature expand_extract ;
[0030] Step S35: Feature expatd_extract After the features are added, they are mapped into the final 360° panoramic video quality prediction score using linear mapping, called y pre ;
[0031] Step S36: Set y pre and gt Input loss function, where y pre is the final 360° panoramic video quality prediction score, y gt To obtain the true quality score of the 360° panoramic video, the monotonicity loss, linear fusion loss, and mean square error are calculated. After adding the losses, back propagation is performed to conduct supervised training of the 360° panoramic video quality evaluation model.
[0032] In step S36, the loss function is specifically as follows: the monotonicity loss and the linear fusion loss are calculated by comparing the true quality scores of all 360° panoramic videos in a training batch with the 360° panoramic video quality scores predicted by the model, and the mean square error between the 360° panoramic video quality scores predicted by each model and the true quality scores of the 360° panoramic video is calculated at the same time. Finally, the losses are summed up to obtain the total loss value. The calculation method is as follows:
[0033] L mae =(y gt -y pred ) 2
[0034] L total =L mo +αL li +βL mae
[0035] Where sign(·) represents the sign function, <> represents the inner product of two vectors, and s pred and gt A vector representing all the predictions and true labels in a training batch, Respectively represent s pred and gt The average value of Respectively represent the bith batch and bj batch The model predicted value, Respectively represent the bith batch and bj batchThe true quality score of the 360° panoramic video, ||·||2 represents the square root of the value, L mo represents the monotonicity loss, L li represents the linear fusion loss, L mae represents the mean square error, L total Represents the overall loss of the model.
[0036] In step S32, the Feature is deleted encode The specific method of redundant features is as follows:
[0037] For Video sample A region in the video that passes through rand_crop is represented as Video crop , Video crop Each frame in is represented as In order to preserve the original spatiotemporal information to the maximum extent, we will discard the information of different spatial positions in turn at different frame positions. The calculation method is as follows: A t =c(A t-1 |A t-2 , ..., A1)
[0038] Among them A t express The retained feature map, c(·|·) represents The feature map of The feature map changes cyclically.
[0039] The step S4 is specifically as follows:
[0040] Step S41: The 360° panoramic video to be evaluated for quality is also processed through steps S21 and S22 to obtain a pre-processed 360° panoramic video Video process ;
[0041] Step S42: Video process The corresponding 360° panoramic video quality score is calculated by step S3, and the higher the score is, the better the quality of the corresponding video is.
[0042] Compared with the prior art, the present invention has the following beneficial effects:
[0043] 1. Compared with the existing 360° panoramic video quality evaluation method, the present invention removes the spatiotemporal redundant features in the 360° panoramic video while maintaining the accuracy as much as possible, improves the reasoning speed of the method, and makes it more suitable for actual scenes;
[0044] 2. The present invention innovatively uses a method for removing out-of-control features of videos using temporal and spatial differences. Different from the video cropping method, the present invention only requires 10% of the computational effort of the traditional method and can achieve an accuracy rate equal to or even higher than that of other traditional methods.
[0045] 3. Inspired by the Video ViT model, the present invention reconstructs and trains the 360° panoramic video quality evaluation backbone network, which makes the occupancy rate of system resources such as disk, video memory, and memory lower than that of existing methods, is more suitable for actual scenarios, and compresses computing resource costs.
[0046] The present invention proposes a method for reducing the amount of model calculation by excluding redundant video information, thereby achieving a faster and more accurate 360° panoramic video quality assessment. By deleting the redundant spatiotemporal information in the 360° panoramic video, the quality characteristics of the 360° panoramic video are retained to the maximum extent, so that the efficiency of 360° panoramic video quality assessment can be improved without losing accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:
[0048] Attached Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION
[0049] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0050] As shown in the figure, the lightweight panoramic video quality evaluation method based on spatiotemporal redundancy uses a panoramic video quality evaluation model and includes the following steps:
[0051] Step S1: Collect 360° panoramic videos, organize the videos into data sets and perform preprocessing;
[0052] Step S2: Map the dataset input into an equirectangular projection format and obtain the final model input after uniform sampling with spatiotemporal consistency;
[0053] Step S3: Remove redundant spatiotemporal information from the 360° panoramic video, use the retained features that highly represent the video quality, compare the predicted video quality score obtained by the model with the actual video quality score, and calculate their monotonicity loss, linear fusion loss, and mean square error to further optimize the model;
[0054] Step S4: input the 360° panoramic video to be detected into the panoramic video quality evaluation model, and judge the quality of the 360° panoramic video to be detected by the score predicted by the model.
[0055] In step S1, a 360° panoramic video in a correct video format required for evaluation is captured by a 360° panoramic video capture camera;
[0056] In step S4, the panoramic video quality assessment model is constructed according to the original architecture of the Video ViT model.
[0057] The step S1 is specifically as follows:
[0058] Step S11: Installing video acquisition equipment at the shooting site;
[0059] Step S12: Shoot a 360° panoramic original video and invite subjects to rate the shot 360° panoramic original video. After excluding outliers, take the average value as the true quality score of the current video, called y gt ;
[0060] Step S13: preprocess all 360° panoramic videos in the data set.
[0061] In step S13, the data preprocessing method includes adjusting the image size, image standardization and 360° panoramic original video true score standardization.
[0062] The step S2 is specifically as follows:
[0063] Step S21: The pre-processed 360° panoramic videos of different projection modes are uniformly mapped into an equidistant rectangular projection format, and the obtained video is called Video ERP ;
[0064] Step S22: Perform temporal and spatial uniform sampling on the pre-processed 360° panoramic video. The calculation method is as follows:
[0065]
[0066] Among them, N h and N w Respectively represent the number of samples in the video height and width, H and W represent Video ERP The height and width of rand_crop means to perform a random crop in a given area. h Indicates the height of vi h Positions, vj w Indicates the width of the vj w Locations, Video sample Indicates Video ERp The results obtained after uniform sampling in time and space;
[0067] Step S23: Video sampleInput the 360° panoramic video quality assessment model for further processing.
[0068] In step S3, redundant spatiotemporal information in the 360° panoramic video is removed by simply encoding the model input.
[0069] The step S3 is specifically as follows:
[0070] Step S31: Video sample The input model is simply encoded through convolution, Feature encode =Conv(Video sample ), where Conv represents convolution calculation, Feature encode Indicates the result obtained after encoding;
[0071] Step S32: Feature encode The features obtained by the same rand_crop are deleted, and only one feature is retained at each position to represent the 360° panoramic video quality at that position. The result obtained is called Feature input ;
[0072] Step S33: Feature input The first, second, and third stage feature maps extracted by the self-attention layer are merged into a set of video quality feature maps of the dataset, called Feature extract ;
[0073] Step S34: Feature extract The first, second, and third stage video feature maps are expanded through the linear mapping layer to make Feature extract The features of the first, second, and third stages have the same number of channels, and the expanded video feature map set is called Feature expand_extract ;
[0074] Step S35: Feature expatd_extract After the features are added, they are mapped into the final 360° panoramic video quality prediction score using linear mapping, called y pre ;
[0075] Step S36: Set y pre and gt Input loss function, where y pre is the final 360° panoramic video quality prediction score, y gt To obtain the true quality score of the 360° panoramic video, the monotonicity loss, linear fusion loss, and mean square error are calculated. After adding the losses, back propagation is performed to conduct supervised training of the 360° panoramic video quality evaluation model.
[0076] In step S36, the loss function is specifically as follows: the monotonicity loss and the linear fusion loss are calculated by comparing the true quality scores of all 360° panoramic videos in a training batch with the 360° panoramic video quality scores predicted by the model, and the mean square error between the 360° panoramic video quality scores predicted by each model and the true quality scores of the 360° panoramic video is calculated at the same time. Finally, the losses are summed up to obtain the total loss value. The calculation method is as follows:
[0077] L mae =(y gt -y pred ) 2
[0078] L total =L mo +αL li +βL mae
[0079] Where sign(·) represents the sign function, <> represents the inner product of two vectors, and s pred and gt A vector representing all the predictions and true labels in a training batch, Respectively represent s pred and gt The average value of Respectively represent the bith batch and bj batch The model predicted value, Respectively represent the bith batch and bj batch The true quality score of the 360° panoramic video, ||·||2 represents the square root of the value, L mo represents the monotonicity loss, L li represents the linear fusion loss, L mae represents the mean square error, L total Represents the overall loss of the model.
[0080] In step S32, the Feature is deleted encode The specific method of redundant features is as follows:
[0081] For Video sample A region in the video that passes through rand_crop is represented as Video crop , Video crop Each frame in is represented as In order to preserve the original spatiotemporal information to the maximum extent, we will discard the information of different spatial positions in turn at different frame positions. The calculation method is as follows: At =c(A t-1 |A t-2 , ..., A1)
[0082] Among them A t express The retained feature map, c(·|·) represents The feature map of The feature map changes cyclically.
[0083] The step S4 is specifically as follows:
[0084] Step S41: The 360° panoramic video to be evaluated for quality is also processed through steps S21 and S22 to obtain a pre-processed 360° panoramic video Video process ;
[0085] Step S42: Video process The corresponding 360° panoramic video quality score is calculated by step S3, and the higher the score is, the better the quality of the corresponding video is.
[0086] In this example, the panoramic video quality assessment model is rebuilt and trained based on the original architecture of the Video ViT model.
[0087] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.
Claims
1. A lightweight panoramic video quality evaluation method based on spatiotemporal redundancy uses a panoramic video quality evaluation model, which is characterized by: The following steps are involved: Step S1: Collect 360° panoramic videos, organize the videos into data sets and perform preprocessing; Step S2: Map the dataset input into an equirectangular projection format and obtain the final model input after uniform sampling with spatiotemporal consistency; Step S3: Remove redundant spatiotemporal information from the 360° panoramic video, use the retained features that highly represent the video quality, compare the predicted video quality score obtained by the model with the actual video quality score, and calculate their monotonicity loss, linear fusion loss, and mean square error to further optimize the model; Step S4: input the 360° panoramic video to be detected into the panoramic video quality evaluation model, and judge the quality of the 360° panoramic video to be detected by the score predicted by the model.
2. The lightweight panoramic video quality assessment method based on spatiotemporal redundancy according to claim 1 is characterized in that: In step S1, a 360° panoramic video in a correct video format required for evaluation is captured by a 360° panoramic video capture camera; In step S4, the panoramic video quality assessment model is constructed according to the original architecture of the VideoViT model.
3. The lightweight panoramic video quality assessment method based on spatiotemporal redundancy according to claim 1 is characterized in that: The step S1 is specifically as follows: Step S11: Installing video acquisition equipment at the shooting site; Step S12: Shoot a 360° panoramic original video and invite subjects to rate the shot 360° panoramic original video. After excluding outliers, take the average value as the true quality score of the current video, called y gt ; Step S13: preprocess all 360° panoramic videos in the data set.
4. The lightweight panoramic video quality assessment method based on spatiotemporal redundancy according to claim 3 is characterized in that: In step S13, the data preprocessing method includes adjusting the image size, image standardization and 360° panoramic original video true score standardization.
5. The lightweight panoramic video quality assessment method based on spatiotemporal redundancy according to claim 1 is characterized in that: The step S2 is specifically as follows: Step S21: The pre-processed 360° panoramic videos of different projection modes are uniformly mapped into an equidistant rectangular projection format, and the obtained video is called Video ERP ; Step S22: Perform temporal and spatial uniform sampling on the pre-processed 360° panoramic video. The calculation method is as follows: Among them, N h and N w Respectively represent the number of samples in the video height and width, H and W represent Video ERP The height and width of rand_crop means to perform a random crop in a given area. h Indicates the height of vi h Positions, vj w Indicates the width of the vj w Locations, Video sample Indicates Video ERP The results obtained after uniform sampling in time and space; Step S23: Video sample Input the 360° panoramic video quality assessment model for further processing.
6. The lightweight panoramic video quality assessment method based on spatiotemporal redundancy according to claim 1 is characterized in that: In step S3, redundant spatiotemporal information in the 360° panoramic video is removed by simply encoding the model input.
7. The lightweight panoramic video quality assessment method based on spatiotemporal redundancy according to claim 1, characterized in that: The step S3 is specifically as follows: Step S31: Video sample The input model is simply encoded through convolution, Feature encode =Conv(Video sample ), where Conv represents convolution calculation, Feature encode Indicates the result obtained after encoding; Step S32: Feature encode The features obtained by the same rand_crop are deleted, and only one feature is retained at each position to represent the 360° panoramic video quality at that position. The result obtained is called Feature input ; Step S33: Feature input The first, second, and third stage feature maps extracted by the self-attention layer are merged into a set of video quality feature maps of the dataset, called Feature extract ; Step S34: Feature extract The first, second, and third stage video feature maps are expanded through the linear mapping layer to make Feature extract The features of the first, second, and third stages have the same number of channels, and the expanded video feature map set is called Feature expand_extract ; Step S35: Feature expand_extract After the features are added, they are mapped into the final 360° panoramic video quality prediction score using linear mapping, called y pre ; Step S36: Set y pre and gt Input loss function, where y pre is the final 360° panoramic video quality prediction score, y gt To obtain the true quality score of the 360° panoramic video, the monotonicity loss, linear fusion loss, and mean square error are calculated. After adding the losses, back propagation is performed to conduct supervised training of the 360° panoramic video quality evaluation model.
8. The lightweight panoramic video quality assessment method based on spatiotemporal redundancy according to claim 7 is characterized in that: In step S36, the loss function is specifically as follows: the monotonicity loss and the linear fusion loss are calculated by comparing the true quality scores of all 360° panoramic videos in a training batch with the 360° panoramic video quality scores predicted by the model, and the mean square error between the 360° panoramic video quality scores predicted by each model and the true quality scores of the 360° panoramic video is calculated at the same time. Finally, the losses are summed up to obtain the total loss value. The calculation method is as follows: L mae (and gt -and pred ) 2 L total =L mo +αL li +βL mae Where sign(·) represents the sign function, <> represents the inner product of two vectors, and s pred and gt A vector representing all the predictions and true labels in a training batch, Respectively represent s pred and gt The average value, Respectively represent the bith batch and bj batch The model predicted value, Respectively represent the bith batch and bj batch The true quality score of the 360° panoramic video, ||·||2 represents the square root of the value, L mo represents the monotonicity loss, L li represents the linear fusion loss, L mae represents the mean square error, L total Represents the overall loss of the model.
9. The lightweight panoramic video quality assessment method based on spatiotemporal redundancy according to claim 7, characterized in that: In step S32, the Feature is deleted encode The specific method of redundant features is as follows: For Video sample A region in the , which passes through rand_crop, is denoted as Vodeo crop , Video crop Each frame in is represented as In order to preserve the original spatiotemporal information to the maximum extent, we will discard the information of different spatial positions in turn at different frame positions. The calculation method is as follows: A t =c(A t-1 |A t-2 ,...,A1) Among them A t express The retained feature map, c(·|·) represents The feature map of The feature map changes cyclically.
10. The lightweight panoramic video quality assessment method based on spatiotemporal redundancy according to claim 1, characterized in that: The step S4 is specifically as follows: Step S41: The 360° panoramic video to be evaluated for quality is also processed through steps S21 and S22 to obtain a pre-processed 360° panoramic video Video process ; Step S42: Video process The corresponding 360° panoramic video quality score is calculated by step S3, and the higher the score is, the better the quality of the corresponding video is.
