Stereoscopic panoramic video comfort prediction method based on deep learning

Abstract

The invention discloses a stereoscopic panoramic video comfort level prediction method based on deep learning. The method comprises the following steps: firstly, randomly playing a three-dimensional panoramic video to a volunteer, and collecting a comfort level score of the volunteer; then acquiring a plane view corresponding to the stereoscopic panoramic video, extracting parallax information, optical flow information, spatial domain information and time domain information from the plane view, fusing the information as an evaluation sample, and taking a corresponding comfort score as an evaluation label; and training the image feature extraction model to predict the shielded content by using the randomly shielded plane image, and completing pre-training. And inputting the evaluation sample into a pre-trained image feature extraction model to complete preliminary feature extraction, inputting a time sequence neural network model to obtain a comfort prediction score of the stereoscopic panoramic video, comparing the comfort prediction score with an evaluation label, and optimizing parameters of the image feature extraction model and the time sequence neural network model. And using the optimized network model to predict the comfort level of the stereoscopic panoramic video.

Description

Technical Field

[0001] The present invention belongs to the field of image vision technology, and relates to the prediction of visual comfort of stereoscopic panoramic videos, and specifically to a method for predicting comfort of stereoscopic panoramic videos based on deep learning. Background Art

[0002] The common modern stereoscopic video / image experience quality evaluation methods mainly focus on the impact of stereoscopic video / image content distortion on quality perception and depth perception, that is, they solve the evaluation problem of the degree of content distortion, but less consider the impact of the stereoscopic video content itself and the impact of visual discomfort on the stereoscopic video / image visual experience quality. The stereoscopic video / image experience quality evaluation model corresponding to the visual characteristics of the human eye should comprehensively consider the impact of multiple visual perception factors on the viewer's visual experience, including left and right viewpoint image quality, depth perception, visual comfort, immersion, and realism. Many solutions have been proposed at home and abroad for the problem of stereoscopic panoramic video comfort.

[0003] Evaluation methods based on disparity and depth information evaluate visual comfort by analyzing the disparity and depth information of stereo images. For example, crosstalk perception and depth estimation are combined with the SSIM model, or predictions are made through the relationship model between features such as disparity amplitude and disparity gradient and subjective evaluation. However, these methods usually rely on the accuracy of disparity maps and depth maps, and are prone to errors in complex scenes or occluded areas. In addition, due to the high complexity of disparity calculation and the limitations of experimental data, the generalization ability of the method is limited to a certain extent.

[0004] Evaluation methods based on saliency and visual attention, based on the human visual attention mechanism, use saliency detection combined with depth or motion information to extract salient areas, and assign higher weights to these areas to evaluate comfort. For example, the SMDE method combines saliency, motion, and depth cues for analysis, or constructs a comfort evaluation model by combining saliency maps with disparity features. This type of method can capture the characteristics of the visual focus area, but the parameter tuning of the saliency area and the complex calculation steps increase the difficulty of implementing the method.

[0005] Comfort evaluation methods based on motion features focus on the impact of motion on visual discomfort and analyze motion features in the horizontal, vertical, and depth directions. For example, by quantifying the motion speed and viewing time of significant objects, or combining temporal and spatial features to perform statistical analysis on overall visual comfort. These methods can reflect the impact of motion on visual fatigue, but often require complex feature extraction and high computational costs.

[0006] Evaluation methods based on machine learning and deep learning use neural networks and machine learning algorithms to predict visual comfort. For example, binocular features of stereo images are learned through binocular fusion networks (BFN), or support vector regression (SVR) is used to build a comfort prediction model. These methods can efficiently capture complex features and establish a mapping relationship with subjective evaluation, but due to the limitation of training data and computational complexity, the generalization ability and real-time performance of the model are still challenges.

[0007] Comfort optimization methods based on reinforcement learning and dynamic adjustment formalize the visual comfort problem as an optimization problem, such as adjusting the parallax range or predicting the user's field of view (FoV) through a reinforcement learning agent model. These methods can dynamically optimize comfort and resource utilization efficiency, but they face high computational costs and device performance limitations in practical applications.

[0008] In summary, it is very important to design a stereoscopic panoramic video comfort evaluation method that can perform well on a variety of different data sets while ensuring that the computational cost is not too high, and achieves versatility, high performance, and high precision. Summary of the invention

[0009] In view of the shortcomings of the existing technology, the present invention proposes a stereoscopic panoramic video comfort prediction method based on deep learning. By utilizing an efficient deep learning model and combining multi-feature fusion technology to analyze video content, parallax, motion and other factors, the accurate prediction of the comfort of the stereoscopic panoramic video is achieved. This optimizes the stereoscopic panoramic video system and improves the user's viewing experience. In addition, the method also takes into account the visual lag effect and input normalization, solving the problems of the existing single data set, computing power bottlenecks, and insufficient versatility.

[0010] A method for predicting comfort level of a stereoscopic panoramic video based on deep learning specifically comprises the following steps: Step 1: Collect stereoscopic panoramic videos with different picture contents, and use a head-mounted VR device to randomly play each stereoscopic panoramic video to volunteers.

[0011] Step 2: Volunteers use the ACR-HR subjective evaluation method to rate the stereoscopic panoramic videos they watch. Collect the ratings of all volunteers, filter outliers, ignore invalid ratings, and get the comfort rating of each video.

[0012] Step 3: Taking the shooting position of the panoramic video as the origin of the coordinate system, assuming that the stereoscopic panoramic video information is distributed on a viewing sphere with a unit radius centered at point O, the viewport is modeled as a plane tangent to the viewing sphere, and the center of the viewport is A pinhole camera model is used to determine the pixels projected from the viewing sphere into the viewport, resulting in a planar view of the stereoscopic panoramic video.

[0013] In order to achieve uniform distribution of pixels in the viewport, it is proposed to use the texture coordinates of the viewport, that is, the center of the viewport Start, then perform the reverse mapping, calculating the point in the viewport The corresponding coordinates on the viewing sphere : in, represents the internal matrix, and Respectively represent the left and right eye focal lengths of the head-mounted VR device, in pixels. , ) is the viewport center R represents the rotation of the volunteer's head position (x', y', z') compared to the standard position (x, y, z), which is the position when looking in the negative direction of the z axis.

[0014] Step 4: Extract disparity information, optical flow information, spatial information and temporal information from the planar view obtained in step 3.

[0015] Preferably, a disparity map, a disparity angle, an optical flow map, motion information, velocity information, acceleration information, time domain information and spatial domain information are extracted from the plan view obtained in step 3.

[0016] Step 5: Fuse the features extracted in step 4, and use the obtained fused feature map as the evaluation sample, and use the comfort score obtained in step 2 as the evaluation label.

[0017] Step 6: Perform random occlusion processing on the plan view obtained in step 3, input it into an image feature extraction model, train the image feature extraction model to predict the occluded content, and complete the pre-training process of the image feature extraction model.

[0018] Preferably, the image feature extraction model is any one of Transformer framework, RNN, GRU, and LSTM.

[0019] Step 7: Input the evaluation sample obtained in step 5 into the pre-trained image feature extraction model in step 6 to complete the preliminary feature extraction, and then input it into a time series neural network model to obtain the comfort prediction score of the stereoscopic panoramic video. Compare it with the evaluation label in step 5 to optimize the parameters of the image feature extraction model and the time series neural network model.

[0020] Preferably, the temporal neural network model is one or more of RNN, GRU, and LSTM.

[0021] Preferably, the evaluation samples obtained in step 5 are respectively input into the pre-trained image feature extraction model in step 6 and an untrained temporal neural network model, and the feature maps output by the two models are fused to output the comfort prediction score of the stereoscopic panoramic video.

[0022] Step 8: Input the planar view of the stereoscopic panoramic video into the model trained in step 7 to obtain a comfort score corresponding to the stereoscopic panoramic video. The obtained comfort score can be used to guide the adjustment of the stereoscopic panoramic video to improve the user experience.

[0023] The present invention has the following beneficial effects: (1) By analyzing factors such as stereoscopic panoramic video content, optical flow, and parallax, an objective evaluation model corresponding to human visual perception is established, providing useful guidance for the optimization of stereoscopic panoramic video systems; (2) By designing a stereoscopic panoramic video comfort database (SPVCD) and extracting various features from the video, different neural network models (such as recurrent neural networks, convolutional neural networks, and machine learning models based on attention mechanisms) are used to predict the quality or comfort score of the video. A balance is achieved in the accuracy, relevance, and stability of the evaluation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of an objective prediction method for comfort of stereoscopic panoramic videos based on deep learning; Figure 2 Modeling schematics for viewports; Figure 3 This is a flowchart of pre-training the Transformer framework in the embodiment.

[0025] Figure 4 It is a method for fusing the output features of the image feature extraction model and the temporal neural network model in the embodiment. DETAILED DESCRIPTION

[0026] The present invention will be further explained below with reference to the accompanying drawings; An objective prediction method for comfort of stereoscopic panoramic video based on deep learning, such as Figure 1 As shown, the specific steps include: Step 1. A total of 116 stereoscopic panoramic videos were collected from the Stanford dataset, the VRQ-TJU dataset, Panocam3d and YouTube. The duration of the video content is 20 seconds, the frame rate ranges from 30fps to 90fps, and the resolution ranges from 4K to 6K. According to the motion characteristics, they can be divided into violent motion, smooth motion and almost still. According to the size of the parallax, they can be divided into large parallax, medium parallax and small parallax. According to the depicted scene, they can be divided into indoor, outdoor and game environments. In order to ensure the balance of the dataset, the number of videos in different categories is kept as consistent as possible.

[0027] The HTC Vive Pro headset was used for video playback, which has a single-eye resolution of 1440×1600, a field of view of 110 degrees, and a refresh rate of 90Hz.

[0028] Step 2: Recruit 30 volunteers to participate in the subjective scoring of stereoscopic panoramic videos, including 12 females and 18 males, aged between 22 and 32 years old. Based on the volunteers' previous experience with VR devices, they were divided into frequent VR users, occasional VR users, and users who had never used VR devices before.

[0029] The subjective rating method adopted the ACR-HR method, using a 5-point rating scale ranging from 1 (very uncomfortable) to 5 (very comfortable).

[0030] In the test phase, a total of 116 videos were evaluated, and appropriate rest periods were provided to relieve participants' fatigue. In the test phase, the test videos were displayed in a random order, and the scene content was placed in the center of the field of vision before the experiment to ensure that all participants saw the same scene, and participants were not allowed to look around during the experiment.

[0031] All scores obtained in the experiment were screened out according to the ITU-BT500 standard for outliers, and then the average score given by all volunteers for a stereoscopic panoramic video was used as the subjective score of the stereoscopic panoramic video.

[0032] Step 3: Taking the shooting position of the panoramic video as the origin of the coordinate system, assuming that the stereoscopic panoramic video information is distributed on a viewing sphere with a unit radius centered at point O, the viewport can be modeled as a plane ABCD tangent to the viewing sphere, and the center of the viewport is ,like Figure 2As shown. The pinhole camera model is used to determine the pixels projected from the viewing sphere to the viewport, which means that the planar view is formed by projecting the 3D points on the viewing sphere onto the viewport plane ABCD through a perspective transformation. If the visible area of ​​the viewing sphere is uniformly sampled and then mapped to the plane ABCD by rays passing through these points from the center of the sphere O, the points on the plane ABCD will be non-uniform.

[0033] In order to achieve uniform distribution of pixels in the viewport, it is proposed to use the texture coordinates of the viewport, that is, the center of the viewport First, perform the reverse mapping and define the standard head position (x, y, z) as the position when the user looks down in the negative direction of the z axis. Define an internal matrix ,in and Respectively represent the left and right eye focal lengths of the head-mounted VR device, in pixels. , ) is the viewport center The coordinates of . is a point in the visible area of ​​the sphere, Represents the 2D homogeneous coordinates of the point on the viewport. The forward projection formula at this point is , where w is the scale factor, then the method for reverse mapping from the viewport to the view sphere is:

[0034] Where R represents the rotation matrix of the user's head position (x', y', z') relative to the standard position (x, y, z).

[0035] Step 4: Extract disparity information, optical flow information, spatial information and temporal information from the planar view obtained in step 3.

[0036] This embodiment selects disparity map, disparity angle, optical flow map, motion information, speed information, acceleration information, time domain information and spatial domain information as extracted features.

[0037] Among them, the disparity map acquisition method is the SGBM algorithm.

[0038] The optical flow map is obtained by the OptFlowDeepFlow algorithm, which uses a convolutional neural network to learn the motion pattern in the image and infer the motion between pixels based on the learned features. The calculated optical flow is stored in the RGB image, where the R channel represents the optical flow in the x direction, the G channel represents the optical flow in the y direction, and the B channel is not used. For the convenience of calculation, the original optical flow value is multiplied by 4 and then added to 128.

[0039] The spatial information is calculated by Sobel filter.

[0040] Temporal features are obtained by calculating the pixel differences between consecutive frames of a video.

[0041] Step 5: Fuse the features extracted in step 4, and use the obtained fused feature map as the evaluation sample, and use the comfort score obtained in step 2 as the evaluation label.

[0042] Step 6: Perform random occlusion processing on the plane view obtained in step 3, such as Figure 3 As shown, the content of each occlusion is one quarter of the original image area, and the occlusion positions are located at the upper left corner, upper right corner, lower left corner, lower right corner and the center.

[0043] This embodiment uses the Transformer framework as the image feature extraction model, inputs the occluded image into the Transformer framework, uses the image before occlusion as the corresponding label, optimizes parameters through the gradient descent algorithm, and trains the Transformer framework to predict the image content of the occluded part. When the model reaches a convergence state, the pre-training of the Transformer framework is completed.

[0044] Step 7: A cross-validation strategy was used to ensure the stability and reliability of the model. The data set was divided into multiple subsets, each of which was used as a test set in turn, and the rest as a training set, so as to evaluate the performance of the model under different data distributions.

[0045] The evaluation samples in the training set are input into the pre-trained Transformer framework in step 6 to complete the preliminary feature extraction, and then input into a time series neural network model to obtain the comfort prediction score of the stereoscopic panoramic video. The evaluation labels corresponding to the evaluation samples are compared, and the difference between the video feature map and the actual comfort score is measured by the loss function. The image feature extraction model and the time series neural network model parameters are optimized using the gradient descent algorithm and back propagation technology, so that the model can accurately predict the comfort score of the video.

[0046] In each iteration of the model, after the gradient is calculated, the learning rate is dynamically adjusted through the learning rate scheduler. In the initial iteration, a larger learning rate is used for rapid convergence. In the later stage, the learning rate is reduced to fine-tune the model parameters to accelerate the convergence of the model and improve the prediction performance of the model. The weight decay (Weight Decay) regularization technology is used to avoid excessive model parameters and improve the generalization ability of the model. The early stopping strategy is designed to stop training when the performance on the validation set has not improved significantly in multiple consecutive epochs to prevent overfitting.

[0047] The temporal neural network model is one of a recurrent neural network (RNN), a gated recurrent unit (GRU) and a long short-term memory network (LSTM).

[0048] In one embodiment, the preliminary features output by the pre-trained Transformer framework are respectively input into multiple temporal neural network models, and the prediction scores output by the multiple temporal neural network models are weighted averaged to obtain the final comfort prediction score of the stereoscopic panoramic video.

[0049] In one embodiment, the evaluation samples in the training set are respectively input into the pre-trained Transformer framework in step 6 and an untrained temporal neural network model, and the feature maps output by the two models are fused to output the comfort prediction score of the stereoscopic panoramic video, such as Figure 4 shown.

[0050] Step 8. After the model training is completed, the test set data is input into the model to evaluate the performance of the model from the two aspects of correlation and error. The correlation index is evaluated using PLCC (Pearson Linear Correlation Coefficient), SROCC (Spearman Rank-order Correlation Coefficient), and KROCC (Kendall Rank-order Correlation Coefficient). The error index is evaluated using RMSE (Root Mean Square Error) and MAE (Mean Absolute Error): Where X represents the evaluation label, Y represents the comfort prediction score output by the model, Cov(X, Y) is the covariance between X and Y, D(X) is the variance of X, and D(Y) is the variance of Y.

[0051] Arrange the subjective evaluation values ​​and objective model prediction values ​​in ascending order. represents the i-th evaluation label after sorting, represents the i-th comfort prediction score, and N represents the total number of videos input into the model.

[0052] in, and They represent the i-th evaluation label and comfort prediction score respectively, and sgn is the symbolic function.

[0053] Here, || means finding the absolute value.

[0054] It should be noted that the contents not described in detail in this specification belong to the prior art known to the professional and technical personnel in this field. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for the technical personnel in this field to modify the technical solutions recorded in the aforementioned embodiments, or to replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for predicting the comfort level of a stereoscopic panoramic video based on deep learning, which collects the comfort scores given by volunteers when watching different stereoscopic panoramic videos, learns the score of each stereoscopic panoramic video through a neural network model, and realizes the prediction of the comfort level of the stereoscopic panoramic video, characterized by: Step 1: Establish a visual sphere model of the panoramic video, use a pinhole camera model to determine the pixels projected from the visual sphere to the viewport, and obtain a planar view of the stereoscopic panoramic video; Step 2: extracting disparity information, optical flow information, spatial information and temporal information from the planar view; Step 3: fuse the multiple features extracted in step 2, and use the obtained fused feature map as an evaluation sample, and use the comfort rating given by the volunteers when watching the corresponding stereoscopic panoramic video as the evaluation label; Step 4: Perform random occlusion processing on the plan view obtained in step 1, input it into an image feature extraction model, train the image feature extraction model to predict the occlusion content, and complete the pre-training process of the image feature extraction model; Step 5: Input the evaluation sample into the pre-trained image feature extraction model to complete the preliminary feature extraction, and then input it into a time series neural network model to obtain the comfort prediction score of the stereoscopic panoramic video; compare it with the evaluation label to optimize the parameters of the image feature extraction model and the time series neural network model; Step 6: Input the planar view of the stereoscopic panoramic video into the model trained in step 5 to obtain a comfort score corresponding to the stereoscopic panoramic video.

2. The method for predicting comfort level of a stereoscopic panoramic video based on deep learning as claimed in claim 1, characterized in that: Collect stereoscopic panoramic videos with different screen contents, and use head-mounted VR devices to randomly play the stereoscopic panoramic videos to volunteers; volunteers use the ACR-HR subjective evaluation method to score the stereoscopic panoramic videos they watched; Collect all volunteers' ratings, filter outliers, ignore invalid ratings, and get the comfort score of each video.

3. The method for predicting comfort level of a stereoscopic panoramic video based on deep learning as claimed in claim 1, characterized in that: Taking the shooting position of the panoramic video as the origin of the coordinate system, the stereoscopic panoramic video information is distributed on a viewing sphere with a unit radius centered at point O. The viewport is modeled as a plane tangent to the viewing sphere, with its center being ; From the center of the viewport To begin, perform the reverse mapping and calculate the point in the viewport The corresponding coordinates on the viewing sphere : in, represents the internal matrix, and Respectively represent the left and right eye focal lengths of the head-mounted VR device; ( , ) is the viewport center Coordinates; R represents the rotation of the volunteer's head direction position (x', y', z') compared to the standard position (x, y, z), where the standard position is the position when looking in the negative direction of the z-axis.

4. The method for predicting comfort level of a stereoscopic panoramic video based on deep learning as claimed in claim 1, characterized in that: Extract disparity map, disparity angle, optical flow map, motion information, velocity information, acceleration information, temporal information and spatial information from the plan view.

5. A method for predicting comfort level of a stereoscopic panoramic video based on deep learning as claimed in claim 4, characterized in that: The disparity map is obtained by the SGBM algorithm, the optical flow map is obtained by the OptFlowDeepFlow algorithm, the spatial information is calculated by the Sobel filter; the temporal features are obtained by calculating the pixel differences between consecutive frames of the video.

6. The method for predicting comfort level of a stereoscopic panoramic video based on deep learning as claimed in claim 1, characterized in that: The image feature extraction model is any one of Transformer framework, RNN, GRU, and LSTM.

7. The method for predicting comfort level of a stereoscopic panoramic video based on deep learning as claimed in claim 1, characterized in that: The temporal neural network model is one or more of RNN, GRU, and LSTM.

8. A method for predicting comfort level of a stereoscopic panoramic video based on deep learning as claimed in any one of claims 1, 6 and 7, characterized in that: The evaluation samples are input into a pre-trained image feature extraction model and an untrained temporal neural network model respectively. The feature maps output by the two models are fused to output the comfort prediction score of the stereoscopic panoramic video.

9. A method for predicting comfort level of a stereoscopic panoramic video based on deep learning as claimed in any one of claims 1, 6 and 7, characterized in that: The preliminary features output by the image feature extraction model are input into multiple time series neural network models respectively, and the prediction scores output by multiple time series neural network models are weighted averaged to obtain the final comfort prediction score of the stereoscopic panoramic video.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 7.

Images