360-degree video viewport trajectory prediction method based on hybrid neural network model

Abstract

The invention discloses a 360-degree video viewport trajectory prediction method based on a hybrid neural network model, and belongs to the field of 360-degree video transmission and trajectory prediction. The method comprises the following steps: firstly, extracting viewport position data at a user side by equipment; then, inputting user viewport position information and a saliency image provided by a server into an encoder for processing, and generating historical input features of a user track; and finally, inputting the historical input features, the viewport position information of the current user and the future saliency image into a decoder, and predicting the viewport trajectory of the user in a period of time in the future. According to the method provided by the invention, the precision of 360-degree video viewport trajectory prediction is improved, and then the user experience of 360-degree videos is improved.

Description

Technical Field

[0001] The present invention relates to the field of 360-degree video transmission and trajectory prediction, and in particular to a 360-degree video viewport trajectory prediction method based on a hybrid neural network model. Background Art

[0002] With the rapid development of virtual reality (VR) technology, 360-degree video, as a field with strong technical comprehensiveness, has gradually become a research hotspot for countries, enterprises and universities. Compared with traditional 2D video, 360-degree video provides a stronger sense of immersion and interactive experience. However, this also brings higher bandwidth requirements to ensure that viewers get good viewing quality.

[0003] When watching 360-degree videos using a head-mounted display, users can only see part of the content within the viewport area of ​​the display, rather than the entire 360-degree video. This requires an effective bandwidth utilization strategy in video transmission. 360-degree video transmission based on viewport prediction is considered a key method to alleviate the pressure of panoramic video on network transmission. Through an accurate and effective viewport prediction model, combined with the user's viewing behavior and the significant features of the video, the user's field of view area in the next time period can be predicted. This method enables high-quality streaming media transmission only within the user's viewport, thereby reducing bandwidth consumption and improving the quality of user experience.

[0004] In viewport prediction of 360-degree videos, two methods are usually used: content-independent methods and content-based methods. Content-independent methods use the historical head movement trajectory of the audience for short-term prediction and perform well, but their performance tends to drop rapidly in long-term prediction. In contrast, content-based viewport prediction methods combine video content to predict future viewports. One common method is to generate a saliency map by detecting salient features in video frames. This method usually performs better than viewport prediction based solely on head trajectory in long-term prediction due to the correlation between the audience's viewing behavior and the salient features of the video.

[0005] Accurately predicting the viewer's viewport area enables high-resolution slice transmission, ensuring that the viewer feels no difference from the full video in terms of experience. However, incorrect viewport prediction may cause the viewer to be unable to see any picture, which significantly reduces the viewing experience. Therefore, building a highly accurate viewport prediction model is the key to ensuring the quality of the user's viewing experience of 360-degree videos. Summary of the invention

[0006] The present invention proposes a 360-degree video viewport trajectory prediction method based on a hybrid neural network model, the purpose of which is to solve the problems raised in the above background technology. The method processes the user's historical trajectory data and historical image saliency through an encoder to extract the user's historical input features. Subsequently, the decoder combines the current trajectory data, future image saliency and historical input features to significantly improve the accuracy and precision of the viewport trajectory prediction.

[0007] The technical solution adopted by this method includes the following steps:

[0008] Step 1: Obtain the historical viewport trajectory data of 360-degree video users and the image saliency of the videos watched by users;

[0009] Step 2: injecting the historical viewport trajectory information and the historical image saliency information into the encoder of the hybrid model to obtain the historical input features of the viewport prediction trajectory;

[0010] Step 3: Inject historical input features, the current user’s viewport position information, and future image saliency into the decoder of the hybrid model to obtain the viewport trajectory prediction for a period of time in the future.

[0011] In the viewport prediction trajectory prediction method based on the hybrid neural network prediction model, in step 1, the user trajectory data includes the pitch angle φ and yaw angle γ of the user's head within a certain time period M, that is, the user's head coordinate V t =(φ t ,γ t ), and the saliency P of each frame of the video watched by the user t , and convert the user's pitch angle and yaw angle in the past M seconds into coordinates in the Cartesian coordinate system

[0012] In the viewport prediction method based on the hybrid neural network prediction model as described above, in step 2, a 360-degree video viewport prediction model is designed, which consists of an encoder and a decoder. The encoder is used to extract the feature value between the user's historical trajectory information and the image saliency information, and the decoder combines the feature value provided by the encoder to process the current user's viewport position and the image saliency features in the future period of time to obtain the viewport trajectory information in the future period of time.

[0013] According to the 360-degree video viewport trajectory prediction method based on the hybrid neural network model of claim 1, the LSTM layer of the encoder in step 2 respectively processes the historical trajectory data of the user and user history image saliency Perform feature extraction, the feature extraction and state preservation process of historical input information are as follows:

[0014]

[0015]

[0016] in They represent the user’s historical viewport trajectory features and the features of the salient image of the video in the past M seconds after LSTM processing. It represents the state information at the current time t after LSTM processes the input information, which is used to save the long-term relationship of data. 0 is the initial state of LSTM. In the encoder, the initial state of LSTM is 0. It is represented as the feature vector obtained by combining the obtained trajectory features and image features in the feature vector dimension.

[0017] The input encoding layer of the hybrid neural network encoder described above combines the trajectory data features and image saliency features in dimensions and inserts the temporal information as follows:

[0018]

[0019]

[0020] Where pos represents the position and i represents the dimension. That is, each dimension of the position encoding corresponds to a sine wave. The wavelength forms a geometric series from 2π to 1000·2π.

[0021] As described above, the self-attention mechanism layer of the hybrid neural network encoder converts the features of the input encoding layer into query vectors, key vectors, and value vectors, calculates the attention value, and focuses on the key factors of the viewport trajectory movement. The feature vector after combining the trajectory features and the image features To perform the conversion, first, the linear transformation matrix W Q , W K , W V Multiplying with the input feature vector, we get the query vector Q, key vector K, and value vector V. The calculation and aggregation process of the self-attention value of this layer is as follows:

[0022]

[0023] The self-attention value can be thought of as the matrix V multiplied by the weights computed by the matrices Q and K. The weights are defined by how each element of the sequence Q is affected by all the other elements in the sequence k. In addition, the softmax function normalizes the weights to produce a distribution between 0 and 1. These weights are then applied to all elements in the sequence V. The scaling factor This is to avoid the problem of too large inner product value when the dimension is too high.

[0024] The operation process of the multi-head attention mechanism can be regarded as repeatedly performing linear projections on Q, K, and V. This multi-head attention mechanism helps the model learn from different representations of Q, K, and V by jointly paying attention to the information of different representation subspaces at other positions. Its operation process is as follows:

[0025] MultiHead(Q,K,V)=Concat(h 1 ,...,h n )

[0026] where h i represents the attention value of each attention head after splitting the input feature, and Concat(·) represents the process of connecting the outputs of each attention head along the dimension of the feature vector. The final historical input feature obtained by the multi-head attention mechanism is represented as O enc

[0027] As described above, the feedforward network layer of the hybrid neural network encoder performs nonlinear changes on the input features obtained by aggregating the attention values ​​of the self-attention mechanism layer to reveal the relationship between different dimensions and calculate the eigenvalues ​​of historical input information. After the multi-head attention layer aggregates all the input features, the feedforward network layer is responsible for performing nonlinear changes on the input features to consider the interactions between different dimensions and capture the complex patterns and relationships in the data. The feedforward layer consists of two linear transformations separated by a nonlinear activation function. The calculation process of this layer is as follows:

[0028] FFN(x)=ReLU(xW 1 +b 1 )W 2 +b 2

[0029] Among them, W 1 , W 2 are two linear matrices, b 1 、b 2 It is W 1 , W 2 The corresponding bias matrix.

[0030] The LSTM layer of the hybrid neural network decoder described above converts the state information obtained by the LSTM layer of the encoder Inject and extract the current user's trajectory information V t and image saliency features of future video frames to be viewed Since the future trajectory information is unknown, the future trajectory information is input Set to 0 to obtain the user trajectory data with the same time dimension as the salient feature Then the future trajectory data and image saliency features Combine to get future input features

[0031] Based on the future input feature value The masked self-attention mechanism layer of the hybrid neural network decoder is used. The calculation method of its self-attention value is the same as that of the self-attention mechanism layer. The difference is that the future trajectory information is masked during the model training process to prevent the use of future trajectory information in the prediction process. The self-attention mechanism layer of the decoder uses the historical input feature value O obtained by the encoder. enc As the key vector and value vector, the input features obtained by the masked self-attention mechanism layer are used as the query vector to calculate the multi-head attention value, and the results are aggregated to obtain the final self-attention value.

[0032] Based on the self-attention value, the input features obtained by aggregating the attention values ​​of the self-attention mechanism layer are nonlinearly changed to reveal the relationship between different dimensions and calculate the future user viewport trajectory prediction feature value

[0033] Based on the user viewport trajectory prediction feature value obtained by the feedforward network layer of the hybrid neural network decoder, the viewport trajectory prediction feature value is converted into the user's three-dimensional coordinate data in the Cartesian coordinate system using the linear model of the fully connected layer to obtain the user's viewport trajectory prediction.

[0034] Based on the encoder and decoder, a viewport prediction model is constructed, and the model is trained using the collected user viewport trajectory data and video saliency, and the model parameters are optimized to learn the potential rules of the user's viewport movement when watching a 360-degree video. The trained model is deployed to the 360-degree video user terminal to predict the user's viewport trajectory.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The method of the present invention first monitors the head state of the user through the user's head-mounted device, obtains the user's viewport trajectory data during the video viewing process, and then inputs the user trajectory data information and the video's saliency feature information into the hybrid neural network model through the saliency of each frame image in the video, thereby realizing the user's future viewport trajectory prediction, making the present invention suitable for the efficient transmission of 360-degree video streams. Finally, by transmitting high-resolution viewport slices of the viewport area to the user, the effect of the user watching a 360-degree video is achieved, thereby improving the user experience quality in immersive video. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a step diagram of the present invention.

[0038] Figure 2 It is the hybrid neural network model of the present invention. DETAILED DESCRIPTION

[0039] In order to clarify the technical problems, technical solutions, implementation processes and performance demonstrations, the present invention is further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings do not have to be drawn to scale unless otherwise specified.

[0040] The present invention proposes a 360-degree video viewport trajectory prediction method based on a hybrid neural network model, the purpose of which is to solve the problems raised in the above background technology. The method processes the user's historical trajectory data and historical image saliency through an encoder to extract the user's historical input features. Subsequently, the decoder combines the current trajectory data, future image saliency and historical input features to significantly improve the accuracy and precision of the viewport trajectory prediction.

[0041] The technical solution adopted by this method includes the following steps:

[0042] Step 1: Obtain the historical viewport trajectory data of 360-degree video users and the image saliency of the videos watched by users;

[0043] Step 2: injecting the historical viewport trajectory information and the historical image saliency information into the encoder of the hybrid model to obtain the historical input features of the viewport prediction trajectory;

[0044] Step 3: Inject historical input features, the current user’s viewport position information, and future image saliency into the decoder of the hybrid model to obtain the viewport trajectory prediction for a period of time in the future.

[0045] The viewport prediction trajectory prediction method based on the hybrid neural network prediction model as described above, the step 1 comprises: the user trajectory data includes the pitch angle φ and the yaw angle γ of the user's head within a certain time period M, that is, the user head coordinate V t =(φ t ,γ t ), and the saliency P of each frame of the video watched by the user t . And convert the user's pitch angle and yaw angle in the past M seconds into coordinates in the Cartesian coordinate system The user's head coordinates can be detected by the sensor of the head-mounted device, and the video image saliency can be obtained by extracting the saliency of each frame of the video.

[0046] In step 2, a 360-degree video viewport prediction model is designed. The model consists of an encoder and a decoder. The encoder is used to extract the input feature values ​​between the user's historical trajectory information and the image saliency information. The decoder combines the input feature values ​​provided by the encoder to process the user's current viewport position and the image saliency features in the future to obtain the viewport trajectory in the future.

[0047] The LSTM layer of the encoder in step 2 respectively processes the user's historical trajectory data and user history image saliency Perform feature extraction, the feature extraction and state preservation process of historical input information are as follows:

[0048]

[0049]

[0050] in They represent the user’s historical viewport trajectory features and the features of the salient image of the video in the past M seconds after LSTM processing. It represents the state information at the current time t after LSTM processes the input information, which is used to save the long-term relationship of data. 0 is the initial state of LSTM. In the encoder, the initial state of LSTM is 0. Concat(·) represents the process of connecting trajectory features and saliency features along the dimension of feature vector. It is represented as the feature vector obtained by combining the obtained trajectory features and image features in the feature vector dimension.

[0051] The input encoding layer of the hybrid neural network encoder described above combines the trajectory data features and image saliency features in dimensions and inserts the temporal information as follows:

[0052]

[0053]

[0054] Where pos represents the position and i represents the dimension. That is, each dimension of the position encoding corresponds to a sine wave. The wavelength forms a geometric series from 2π to 1000·2π.

[0055] As described above, the self-attention mechanism layer of the hybrid neural network encoder converts the features of the input encoding layer into query vectors, key vectors, and value vectors, calculates the attention value, and focuses on the key factors of the viewport trajectory movement; the calculation and aggregation process of the self-attention value of this layer is as follows:

[0056] The feature vector after combining trajectory features and image features To perform the conversion, first, the linear transformation matrix W Q , W K , W V Multiplying with the input feature vector, we get the query vector Q, key vector K and value vector V.

[0057]

[0058]

[0059] The self-attention value can be thought of as the matrix V multiplied by the weights computed by the matrices Q and K. The weights are defined by how each element of the sequence Q is affected by all the other elements in the sequence k. In addition, the softmax function normalizes the weights to produce a distribution between 0 and 1. These weights are then applied to all elements in the sequence V. The scaling factor This is to avoid the problem of too large inner product value when the dimension is too high.

[0060] The operation process of the multi-head attention mechanism can be regarded as repeatedly performing linear projections on Q, K, and V. This multi-head attention mechanism helps the model learn from different representations of Q, K, and V by jointly paying attention to the information of different representation subspaces at other positions. Its operation process is as follows:

[0061] MultiHead(Q,K,V)=Concat(h 1 ,...,h n )

[0062] Where h i =Attention(Q i ,K i ,V i )

[0063] where h i represents the attention value of each attention head after splitting the input feature, and Concat(·) represents the process of connecting the outputs of each attention head along the dimension of the feature vector. The final historical input feature obtained by the multi-head attention mechanism is represented as O enc

[0064] As described above, the feedforward network layer of the hybrid neural network encoder performs nonlinear changes on the input features obtained by aggregating the attention values ​​of the self-attention mechanism layer to reveal the relationship between different dimensions and calculate the eigenvalues ​​of historical input information. After the multi-head attention layer aggregates all the input features, the feedforward network layer is responsible for performing nonlinear changes on the input features to consider the interactions between different dimensions and capture the complex patterns and relationships in the data. The feedforward layer consists of two linear transformations separated by a nonlinear activation function. The calculation process of this layer is as follows:

[0065] FFN(x)=ReLU(xW 1 +b 1 )W 2 +b 2

[0066] Among them, W 1 , W 2 are two linear matrices, b 1 、b 2 It is W 1 , W 2 The corresponding bias matrix.

[0067] The LSTM layer of the hybrid neural network decoder in step 3 receives the state information obtained by the LSTM layer of the encoder Inject and extract the current user's trajectory information V t and image saliency features of future video frames to be viewed Since the future trajectory information is unknown, the future trajectory information is input Set to 0 to obtain the user trajectory data with the same time dimension as the salient feature Then the future trajectory data and image saliency features Combine to get future input features

[0068] Based on the future input feature value The masked self-attention mechanism layer of the hybrid neural network decoder is used. The calculation method of its self-attention value is the same as that of the self-attention mechanism layer. The difference is that the future trajectory information is masked during the model training process to prevent the use of future trajectory information in the prediction process. The self-attention mechanism layer of the decoder uses the historical input feature value O obtained by the encoder. enc As the key vector and value vector, the input features obtained by the masked self-attention mechanism layer are used as the query vector to calculate the multi-head attention value, and the results are aggregated to obtain the final self-attention value.

[0069] Based on the self-attention value, the input features obtained by aggregating the attention values ​​of the self-attention mechanism layer are nonlinearly changed to reveal the relationship between different dimensions and calculate the future user viewport trajectory prediction feature value

[0070] Based on the user viewport trajectory prediction feature value obtained by the feedforward network layer of the hybrid neural network decoder, the viewport trajectory prediction feature value is converted into the user's three-dimensional coordinate data in the Cartesian coordinate system using the linear model of the fully connected layer to obtain the user's viewport trajectory prediction.

[0071] Based on the encoder and decoder, a viewport prediction model is constructed, and the model is trained using the collected user viewport trajectory data and video saliency, and the model parameters are optimized to learn the potential rules of the user's viewport movement when watching a 360-degree video. The trained model is deployed to the 360-degree video user terminal to predict the user's viewport trajectory.

[0072] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A 360-degree video viewport trajectory prediction method based on a hybrid neural network model, characterized in that: include: Step 1: Obtain the historical viewport trajectory data of 360-degree video users and the image saliency of the videos watched by users; Step 2: injecting the historical viewport trajectory information and the historical image saliency information into the encoder of the hybrid model to obtain the historical input features of the viewport prediction trajectory; Step 3: Inject historical input features, the current user’s viewport position information, and future image saliency into the decoder of the hybrid model to obtain the viewport trajectory prediction for a period of time in the future.

2. The 360-degree video viewport trajectory prediction method based on a hybrid neural network model according to claim 1, characterized in that: The user trajectory data in step 1 includes the pitch angle and yaw angle of the user's head within a certain period of time, and the saliency map of each frame image of the video watched by the user.

3. The 360-degree video viewport trajectory prediction method based on a hybrid neural network model according to claim 1, characterized in that: A 360-degree video viewport prediction model is designed. The model consists of an encoder and a decoder. The encoder is used to extract the feature values ​​between the user's historical trajectory information and the image saliency information. The decoder combines the feature values ​​provided by the encoder to process the current user's viewport position and the image saliency features in the future to obtain the viewport trajectory information in the future.

4. The 360-degree video viewport trajectory prediction method based on a hybrid neural network model according to claim 1, characterized in that: The model encoder in step 2 includes: The LSTM layer is used to extract the user's historical trajectory information and image saliency features, and save the status of historical input information; An input coding layer combines the trajectory data features and the image saliency features in dimensions based on the user history trajectory information and the image saliency features, and inserts the time series information to obtain an input feature containing the time series information; The self-attention mechanism layer converts the features of the input encoding layer into query vectors, key vectors and value vectors based on the input features containing time series information, calculates the attention value, and focuses on the key factors of the viewport trajectory movement; The feedforward network layer, based on the attention value, performs nonlinear changes on the input features obtained by aggregating the attention values ​​of the self-attention mechanism layer to reveal the relationship between different dimensions and calculate the characteristic values ​​of the historical input information.

5. The 360-degree video viewport trajectory prediction method based on a hybrid neural network model according to claim 1, characterized in that: The model decoder converts the input of the encoder's feature values ​​and the current user information into the required viewport trajectory data. The model decoder in step 3 includes: The LSTM layer, based on the input state saved by the encoder, uses the state information as the initial state to extract the current user's trajectory information and the image saliency features of the video frame to be watched in the future; Input encoding layer, which combines trajectory data features and image saliency features in dimension and inserts temporal information; A masked self-attention mechanism layer, based on the trajectory data features and image saliency features obtained by the input encoding layer, masks the future trajectory information during the model training process to prevent the future trajectory information from being used in the prediction process; A self-attention mechanism layer, based on the feature values ​​obtained by the encoder as key vectors and value vectors, and using the input features obtained by the masked attention mechanism layer as query vectors to calculate the attention value; The feed-forward network layer performs nonlinear changes based on the input features obtained by aggregating the attention values ​​of the self-attention mechanism layer to reveal the relationship between different dimensions and calculate the future user viewport trajectory prediction value; The fully connected layer converts the predicted data dimension obtained by the feedforward network layer into three-dimensional viewport position coordinates to obtain the user's viewport trajectory prediction within a period of time in the future.

Images