A method for predicting the viewing viewpoint of panoramic video streaming with viewer type estimation

Abstract

This invention discloses a method for predicting the viewport of panoramic video streaming with viewer type estimation. This method, tailored to different viewer types, trains a corresponding neural network prediction model for the viewport position using a set of viewport position vector samples. This makes the neural network prediction model more viewer-type specific, ensuring that a particular viewport position prediction model is only applicable to a specific viewer type, reducing the training difficulty of the model and improving prediction accuracy. This method determines the viewer type based on the viewer's actual viewport position sequence before the current frame and selects the corresponding neural network prediction model to predict the viewport position of the current frame. This method ensures that for different types of viewers, a corresponding trained neural network prediction model is used to predict the viewport position of the current frame.

Description

Technical Field

[0001] This invention relates to a method for predicting the viewing viewport in panoramic video streaming with viewer type estimation, belonging to the field of panoramic video streaming and playback technology. Background Technology

[0002] With the continuous improvement of network communication speeds, panoramic video streaming has gradually gained attention and become an important form of digital media presentation. Panoramic videos typically have high resolution, resulting in a large amount of data for streaming over the network, which can easily lead to playback stuttering. It's important to note that although panoramic videos store visual information from a 360-degree horizontal and 180-degree vertical panoramic field of view, viewers using head-mounted displays only see a portion of the panoramic video at any given moment—this portion is the content visible through the viewport. Since the panoramic content outside the viewport is not seen by the viewer, it can be transmitted at a low bitrate or even omitted entirely, without affecting the viewing experience. Therefore, predicting the viewer's viewport position in advance for each frame of the panoramic video during panoramic video streaming is highly valuable on the computer video server.

[0003] The 2022 master's thesis from Xi'an University of Electronic Science and Technology, titled "Viewport Prediction and Adaptive Transmission Technology for 360-Degree Video," introduces a viewport position prediction method based on Long Short-Term Memory (LSTM) networks (see Section 3.2 of the thesis). Influenced by factors such as gender and age, different viewers exhibit significantly different viewport position change trajectories while watching panoramic videos. Predicting the viewport position of all types of viewers using a single trained neural network model is difficult to achieve sufficiently high prediction accuracy. A Chinese invention patent application (application number 202510984666.3) discloses a clustering analysis method for viewport trajectories in panoramic video streaming. This method can cluster the viewport trajectories of different viewers of the same panoramic video and calculate the viewport position sequence corresponding to the average viewport trajectory of different clusters. Using this invention patent application, historical viewport trajectories in panoramic video-on-demand transmission applications can be clustered (each cluster corresponds to a viewer type) to obtain the viewing behavior characteristics of different types of viewers. All viewport position vector samples in each cluster (see the description in Chinese invention patent application No. 202510984666.3) are used as a dataset to train a neural network prediction model for viewport position. This model can predict the viewport position of the current frame based on the actual viewport positions of the frames preceding it. For a new viewer, their viewport position type can be estimated based on their actual viewport position change sequence preceding the current frame, combined with the average viewport trajectory corresponding to different viewer types obtained from cluster analysis. Figure 1 As shown, the viewport position of the viewer in the current frame is predicted based on a viewport position neural network prediction model corresponding to the estimated viewer type. It is worth noting that the number of viewport position sequence samples contained in each cluster obtained from cluster analysis may vary significantly, with some clusters containing a very small number of such samples. Therefore, in this application, if the number of viewport position sequence samples contained in a certain cluster is less than a given threshold, the viewport position sequence samples contained in that cluster are merged into the nearest neighbor cluster whose number of viewport position sequences is greater than or equal to the given threshold. Summary of the Invention

[0004] The purpose of this invention is to provide a panoramic video streaming viewing viewport prediction method with viewer type estimation. This method can determine the viewer type based on the viewer's actual viewing viewport position sequence before the current frame, and then select a viewing viewport position neural network prediction model corresponding to the viewer type to predict the viewing viewport position of the current frame.

[0005] This method is implemented as follows, including the following steps: 1) The first part of this method clusters all viewing viewport position sequences of the panoramic video PanoV stored in the disk storage of the computer server to obtain K clusters and their respective average viewport trajectories. At the same time, it trains the corresponding viewing viewport position neural network prediction model for the clusters to be further processed. The specific steps are as follows: S101. Read all viewing port position sequences of the panoramic video PanoV from the computer server's disk storage and store them in the computer server's memory; each viewing port position sequence contains N viewing port positions, where N is a positive integer; treat each viewing port position sequence as a viewing port position vector sample, and all viewing port position vector samples of the panoramic video PanoV constitute the viewing port position vector sample set DS; use the panoramic video streaming viewing port trajectory clustering analysis method to cluster the viewing port position vector sample set DS, obtaining K clusters and the viewing port position sequence corresponding to the average viewing port trajectory of each cluster; K is a positive integer; S102. Create a two-dimensional array DKK with K rows and K columns in the memory of the computer server, and set the value of all elements of the two-dimensional array DKK to 0; create a two-dimensional array DCC with K rows and K columns in the memory of the computer server, and set the value of all elements of the two-dimensional array DCC to -2. S103. For i = 1, 2, ..., K, perform the following operations in sequence: S103-1, Treat the sequence of viewing viewpoint positions corresponding to the average viewpoint trajectory of the i-th cluster as a viewing viewpoint position vector sample S01; S103-2. For j = 1, 2, ..., K, perform the following operations in sequence: S103-2-1. Treat the sequence of viewing viewpoint positions corresponding to the average viewpoint trajectory of the j-th cluster as a viewing viewpoint position vector sample S02. S103-2-2. Calculate the spherical distance D between the viewing port position vector samples S01 and S02, where D = d_1 + d_2 + ... + d_N, and d_n represents the spherical distance between the points corresponding to the nth viewing port position of each of the viewing port position sequences corresponding to the viewing port position vector samples S01 and S02 on the panoramic imaging sphere, n = 1, 2, ..., N; if the number of viewing port position vector samples contained in the j-th cluster is less than the threshold NC, where NC is a positive integer, then let the spherical distance D be INF, where INF represents the maximum positive value that the elements of the two-dimensional array DKK can store; if i = j, then let the spherical distance D be INF; S103-2-3. Let the value of the element in the i-th row and j-th column of the two-dimensional array DKK be the spherical distance D; S103-2-4, The operation for the j-th cluster ends; S103-3, The operation for the i-th cluster ends; S104. For i = 1, 2, ..., K, perform the following operations in sequence: S104-1. If the number of viewing port position vector samples contained in the i-th cluster is greater than or equal to the threshold NC, proceed to step S104-3; otherwise, calculate the column number Jmin of the element with the smallest value in the i-th row of the two-dimensional array DKK. S104-2. Set the value of the element in the i-th row and Jmin-th column of the two-dimensional array DCC to 1; S104-3, The operation for the i-th cluster ends; S105. For j = 1, 2, ..., K, perform the following operations in sequence: S105-1. If the number of viewing port position vector samples contained in the j-th cluster is less than the threshold NC, proceed to step S105-3. Otherwise, check the elements E in the j-th column of the two-dimensional array DCC one by one. If the value of element E is equal to 1, add all the viewing port position vector samples of the IE-th cluster to the j-th cluster. IE represents the row number of element E in the two-dimensional array DCC. S105-2, Recalculate the viewing position sequence corresponding to the average viewport trajectory of the j-th cluster; S105-3, The operation for the j-th cluster ends; S106. For j = 1, 2, ..., K, perform the following operations in sequence: S106-1. If the number of viewing port position vector samples contained in the j-th cluster is less than the threshold NC, proceed to step S106-2. Otherwise, construct a dataset using the viewing port position sequence corresponding to all viewing port position vector samples contained in the j-th cluster to complete the training of the viewing port position neural network prediction model corresponding to the j-th cluster. S106-2, The operation on the j-th cluster ends.

[0006] 2) The second part of this method involves estimating the viewer type of the current viewer on a computer server, and selecting a corresponding neural network prediction model for the viewport position based on the estimated viewer type to predict the viewport position of the current frame. The specific steps are as follows: S201. Create a list LD containing K elements in the memory of the computer server. Each element of list LD stores a spherical distance value. Let the value of each element of list LD be INF2, where INF2 represents the maximum positive value that the elements of list LD can store. S202. For n1 = 2, 3, ..., N, perform the following operations in sequence: S202-1. Receive the actual viewing port position of the current viewer when watching the n1-1th frame of the panoramic video PanoV, which is transmitted from the client to the computer server, and save it in the memory of the computer server. S202-2. Arrange the actual viewing port positions of the current viewer when watching the panoramic video PanoV from the 1st to the n1-1th frames in ascending order according to their corresponding frame numbers to obtain a sequence containing n1-1 viewing port positions, which is called the viewing port position vector sample A01. S202-3. For j = 1, 2, ..., K, perform the following operations in sequence: S202-3-1. If the number of viewing port position vector samples contained in the j-th cluster is less than the threshold NC, proceed to step S202-3-3. Otherwise, the subsequence consisting of the 1st to the n1-1th viewing port positions of the viewing port position sequence corresponding to the average viewing port trajectory of the j-th cluster is called the viewing port position vector sample B_i. S202-3-2. Calculate the spherical distance D2 between the viewing port position vector sample A01 and the viewing port position vector sample B_i, D2 = d_1 + d_2 + ... + d_(n1-1), where d_t represents the spherical distance between the points corresponding to the t-th viewing port position of the viewing port position sequence corresponding to the viewing port position vector sample A01 and the viewing port position vector sample B_i on the panoramic imaging sphere, t = 1, 2, ..., (n1-1); assign the spherical distance D2 to the j-th element of the list LD; S202-3-3, The operation for the j-th cluster ends; S202-4. Find the index Ik of the smallest element in list LD; S202-5. Estimate the type of the current viewer as the viewer type corresponding to the Ikth cluster; S202-6. Using the neural network prediction model corresponding to the Ikth cluster, predict the viewing position of the current viewer when watching the n1st frame based on the actual viewing position of the current viewer when watching the panoramic video PanoV from the 1st to the n1-1th frames. S202-7, Operations on the n1st frame have ended.

[0007] The positive effects of this invention are as follows: This method, for different types of viewers, can use corresponding viewport position vector sample sets to train corresponding viewport position neural network prediction models. This makes the viewport position neural network prediction models more viewer-type targeted, ensuring that a specific viewport position neural network prediction model is only for a specific viewer type, reducing the training difficulty of the viewport position neural network prediction model and improving prediction accuracy. Furthermore, this method can determine the viewer type based on the viewer's actual viewport position sequence before the current frame (in the aforementioned steps, the n1th frame represents the current frame), and select the viewport position neural network prediction model corresponding to that viewer type to predict the viewport position of the current frame. This method ensures that for different types of viewers, the corresponding trained viewport position neural network prediction model is used to predict the viewport position of the current frame. Attached Figure Description

[0008] Figure 1 A schematic diagram illustrating the viewport prediction overview process. Detailed Implementation

[0009] To make the features and advantages of this method clearer, the method is further described below with reference to specific embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. This embodiment considers panoramic video stored on a computer server. Viewers obtain panoramic video data via streaming over a network on a client and view the panoramic video using a head-mounted display. The client transmits the viewer's actual viewing viewpoint position for each frame of the panoramic video to the computer server in real time. The client transmits an actual viewing viewpoint position to the computer server for each frame.

[0010] This method is implemented as follows, including the following steps: 1) The first part of this method clusters all viewing viewport position sequences of the panoramic video PanoV stored in the disk storage of the computer server to obtain K clusters and their respective average viewport trajectories. At the same time, it trains the corresponding viewing viewport position neural network prediction model for the clusters to be further processed. The specific steps are as follows: S101. Read all viewing port position sequences of the panoramic video PanoV from the computer server's disk storage and store them in the computer server's memory; each viewing port position sequence contains N viewing port positions, where N is a positive integer; treat each viewing port position sequence as a viewing port position vector sample, and all viewing port position vector samples of the panoramic video PanoV constitute the viewing port position vector sample set DS; use the panoramic video streaming viewing port trajectory clustering analysis method to cluster the viewing port position vector sample set DS, obtaining K clusters and the viewing port position sequence corresponding to the average viewing port trajectory of each cluster; K is a positive integer; S102. Create a two-dimensional array DKK with K rows and K columns in the memory of the computer server, and set the value of all elements of the two-dimensional array DKK to 0; create a two-dimensional array DCC with K rows and K columns in the memory of the computer server, and set the value of all elements of the two-dimensional array DCC to -2. S103. For i = 1, 2, ..., K, perform the following operations in sequence: S103-1, Treat the sequence of viewing viewpoint positions corresponding to the average viewpoint trajectory of the i-th cluster as a viewing viewpoint position vector sample S01; S103-2. For j = 1, 2, ..., K, perform the following operations in sequence: S103-2-1. Treat the sequence of viewing viewpoint positions corresponding to the average viewpoint trajectory of the j-th cluster as a viewing viewpoint position vector sample S02. S103-2-2. Calculate the spherical distance D between the viewing port position vector samples S01 and S02, where D = d_1 + d_2 + ... + d_N, and d_n represents the spherical distance between the points corresponding to the nth viewing port position of each of the viewing port position sequences corresponding to the viewing port position vector samples S01 and S02 on the panoramic imaging sphere, n = 1, 2, ..., N; if the number of viewing port position vector samples contained in the j-th cluster is less than the threshold NC, where NC is a positive integer, then let the spherical distance D be INF, where INF represents the maximum positive value that the elements of the two-dimensional array DKK can store; if i = j, then let the spherical distance D be INF; S103-2-3. Let the value of the element in the i-th row and j-th column of the two-dimensional array DKK be the spherical distance D; S103-2-4, The operation for the j-th cluster ends; S103-3, The operation for the i-th cluster ends; S104. For i = 1, 2, ..., K, perform the following operations in sequence: S104-1. If the number of viewing port position vector samples contained in the i-th cluster is greater than or equal to the threshold NC, proceed to step S104-3; otherwise, calculate the column number Jmin of the element with the smallest value in the i-th row of the two-dimensional array DKK. S104-2. Set the value of the element in the i-th row and Jmin-th column of the two-dimensional array DCC to 1; S104-3, The operation for the i-th cluster ends; S105. For j = 1, 2, ..., K, perform the following operations in sequence: S105-1. If the number of viewing port position vector samples contained in the j-th cluster is less than the threshold NC, proceed to step S105-3. Otherwise, check the elements E in the j-th column of the two-dimensional array DCC one by one. If the value of element E is equal to 1, add all the viewing port position vector samples of the IE-th cluster to the j-th cluster. IE represents the row number of element E in the two-dimensional array DCC. S105-2, Recalculate the viewing position sequence corresponding to the average viewport trajectory of the j-th cluster; S105-3, The operation for the j-th cluster ends; S106. For j = 1, 2, ..., K, perform the following operations in sequence: S106-1. If the number of viewing port position vector samples contained in the j-th cluster is less than the threshold NC, proceed to step S106-2. Otherwise, construct a dataset using the viewing port position sequence corresponding to all viewing port position vector samples contained in the j-th cluster to complete the training of the viewing port position neural network prediction model corresponding to the j-th cluster. S106-2, The operation on the j-th cluster ends.

[0011] 2) The second part of this method involves estimating the viewer type of the current viewer on a computer server, and selecting a corresponding neural network prediction model for the viewport position based on the estimated viewer type to predict the viewport position of the current frame. The specific steps are as follows: S201. Create a list LD containing K elements in the memory of the computer server. Each element of list LD stores a spherical distance value. Let the value of each element of list LD be INF2, where INF2 represents the maximum positive value that the elements of list LD can store. S202. For n1 = 2, 3, ..., N, perform the following operations in sequence: S202-1. Receive the actual viewing port position of the current viewer when watching the n1-1th frame of the panoramic video PanoV, which is transmitted from the client to the computer server, and save it in the memory of the computer server. S202-2. Arrange the actual viewing port positions of the current viewer when watching the panoramic video PanoV from the 1st to the n1-1th frames in ascending order according to their corresponding frame numbers to obtain a sequence containing n1-1 viewing port positions, which is called the viewing port position vector sample A01. S202-3. For j = 1, 2, ..., K, perform the following operations in sequence: S202-3-1. If the number of viewing port position vector samples contained in the j-th cluster is less than the threshold NC, proceed to step S202-3-3. Otherwise, the subsequence consisting of the 1st to the n1-1th viewing port positions of the viewing port position sequence corresponding to the average viewing port trajectory of the j-th cluster is called the viewing port position vector sample B_i. S202-3-2. Calculate the spherical distance D2 between the viewing port position vector sample A01 and the viewing port position vector sample B_i, D2 = d_1 + d_2 + ... + d_(n1-1), where d_t represents the spherical distance between the points corresponding to the t-th viewing port position of the viewing port position sequence corresponding to the viewing port position vector sample A01 and the viewing port position vector sample B_i on the panoramic imaging sphere, t = 1, 2, ..., (n1-1); assign the spherical distance D2 to the j-th element of the list LD; S202-3-3, The operation for the j-th cluster ends; S202-4. Find the index Ik of the smallest element in list LD; S202-5. Estimate the type of the current viewer as the viewer type corresponding to the Ikth cluster; S202-6. Using the neural network prediction model corresponding to the Ikth cluster, predict the viewing position of the current viewer when watching the n1st frame based on the actual viewing position of the current viewer when watching the panoramic video PanoV from the 1st to the n1-1th frames. S202-7, Operations on the n1st frame have ended.

[0012] For a detailed description of the panoramic imaging sphere, please refer to Figure 2 and related text description in Chinese Invention Patent Application No. 202510984666.3.

[0013] In this embodiment, NC = 100, N = 10000, and after executing step S101, three or more of the K clusters obtained contain viewing port position vector samples greater than or equal to NC. If the elements of the two-dimensional array DKK are stored in the memory of the computer server using floating-point variables, INF represents the maximum value that the floating-point variable can store. The maximum values ​​that variables of different data types can store are explained in programming courses. In step S105-2, the viewing port position sequence corresponding to the average viewing port trajectory is calculated using the method described in step S4 of Chinese invention patent application number 202510984666.3. Step S4 of Chinese invention patent application number 202510984666.3 calculates the corresponding viewing port position sequence based on the average viewing port trajectory for each of the K clusters obtained in step S3 of that patent application. In step S105-1, if the value of element E is equal to 1, then all viewing port position vector samples of the IE-th cluster are added to the j-th cluster, where IE represents the row number of element E in the two-dimensional array DCC. In computer programming, adding all viewing port position vector samples of the IE-th cluster to the j-th cluster simply involves adding a copy of all viewing port position vector samples from the IE-th cluster to the j-th cluster; the viewing port position vector samples in the IE-th cluster are not deleted. In this embodiment, the viewing port position neural network prediction model uses the LSTM neural network described in the "LSTM-based Viewport Prediction Algorithm" introduced in the 2022 Master's Thesis "Viewport Prediction and Adaptive Transmission Technology for 360-Degree Video" from Xi'an University of Electronic Science and Technology as the viewing port position neural network prediction model. In step S201, INF2 represents the maximum positive value that the elements of list LD can store; if the elements of list LD are stored in the memory of the computer server using floating-point variables, INF2 represents the maximum value that the floating-point variable can store. For the definition of the viewing port position of the panoramic video image, please refer to the description in Chinese invention patent application No. 202510984666.3.

[0014] Each viewing viewpoint position in the viewing viewpoint position sequence corresponds one-to-one with a frame in the panoramic video, and the number of frames in the panoramic video is the same as the number of viewing viewpoint positions in the viewing viewpoint position sequence. In this embodiment, the viewing viewpoint position is predicted starting from the second frame, therefore n1 = 2, 3, ..., N in step S202; the viewing viewpoint position is not predicted for the first frame. In steps S202-6, the viewing viewpoint position neural network prediction model corresponding to the Ikth cluster obtained in step S106 is used to predict the viewing viewpoint position of the current viewer when viewing the n1th frame, based on the actual viewing viewpoint positions of the current viewer when viewing frames 1 to n1-1 of the panoramic video PanoV.

[0015] In this embodiment, the panoramic video streaming player is located on the client side, and the viewer's head-mounted display is connected to the client. If panoramic video streaming is implemented based on DASH (Dynamic Adaptive Streaming over HTTP), the panoramic video PanoV is pre-divided into segments of several seconds in time, and each segment is further divided into several tiles in space. The computer server transmits one segment (divided into several independent tiles) to the client at a time. In this case, before transmitting a segment to the client, the computer server can select an appropriate segment version to send to the client based on the predicted viewport position corresponding to the first frame of that segment. By selecting tile versions with different bitrates to compose segments, different versions of segments are obtained. Note that the first frame of each segment corresponds to a frame of the panoramic video PanoV. In step S202, n1 represents the sequence number of the n1th frame of the panoramic video PanoV.

[0016] In this embodiment, the frame numbers corresponding to the viewport positions in the viewport position sequence are arranged in a forward-to-back order, and each frame number corresponds to only one viewport position. The element numbers in the list LD start from 1, with the first element numbered 1, the second element numbered 2, and so on.

Claims

1. A method for predicting the viewing viewport of panoramic video streaming with viewer type estimation, characterized in that: The technical solution of this method is implemented as follows: Perform the following steps: 1) The first part of this method clusters all viewing viewport position sequences of the panoramic video PanoV stored in the disk storage of the computer server to obtain K clusters and their respective average viewport trajectories. At the same time, it trains the corresponding viewing viewport position neural network prediction model for the clusters to be further processed. The specific steps are as follows: S101. Read all viewing port position sequences of the panoramic video PanoV from the computer server's disk storage and store them in the computer server's memory; each viewing port position sequence contains N viewing port positions, where N is a positive integer; treat each viewing port position sequence as a viewing port position vector sample, and all viewing port position vector samples of the panoramic video PanoV constitute the viewing port position vector sample set DS; use the panoramic video streaming viewing port trajectory clustering analysis method to cluster the viewing port position vector sample set DS, obtaining K clusters and the viewing port position sequence corresponding to the average viewing port trajectory of each cluster; K is a positive integer; S102. Create a two-dimensional array DKK with K rows and K columns in the memory of the computer server, and set the value of all elements of the two-dimensional array DKK to 0; create a two-dimensional array DCC with K rows and K columns in the memory of the computer server, and set the value of all elements of the two-dimensional array DCC to -2. S103. For i = 1, 2, ..., K, perform the following operations in sequence: S103-1, Treat the sequence of viewing viewpoint positions corresponding to the average viewpoint trajectory of the i-th cluster as a viewing viewpoint position vector sample S01; S103-2. For j = 1, 2, ..., K, perform the following operations in sequence: S103-2-1. Treat the sequence of viewing viewpoint positions corresponding to the average viewpoint trajectory of the j-th cluster as a viewing viewpoint position vector sample S02. S103-2-2. Calculate the spherical distance D between the viewing port position vector samples S01 and S02, where D = d_1 + d_2 + ... + d_N, and d_n represents the spherical distance between the points corresponding to the nth viewing port position of each of the viewing port position sequences corresponding to the viewing port position vector samples S01 and S02 on the panoramic imaging sphere, n = 1, 2, ..., N; if the number of viewing port position vector samples contained in the j-th cluster is less than the threshold NC, where NC is a positive integer, then let the spherical distance D be INF, where INF represents the maximum positive value that the elements of the two-dimensional array DKK can store; If i = j, then let the spherical distance D be INF; S103-2-3. Let the value of the element in the i-th row and j-th column of the two-dimensional array DKK be the spherical distance D; S103-2-4, The operation for the j-th cluster ends; S103-3, The operation for the i-th cluster ends; S104. For i = 1, 2, ..., K, perform the following operations in sequence: S104-1. If the number of viewing port position vector samples contained in the i-th cluster is greater than or equal to the threshold NC, proceed to step S104-3; otherwise, calculate the column number Jmin of the element with the smallest value in the i-th row of the two-dimensional array DKK. S104-2. Set the value of the element in the i-th row and Jmin-th column of the two-dimensional array DCC to 1; S104-3, The operation for the i-th cluster ends; S105. For j = 1, 2, ..., K, perform the following operations in sequence: S105-1. If the number of viewing port position vector samples contained in the j-th cluster is less than the threshold NC, proceed to step S105-3. Otherwise, check the elements E in the j-th column of the two-dimensional array DCC one by one. If the value of element E is equal to 1, add all the viewing port position vector samples of the IE-th cluster to the j-th cluster. IE represents the row number of element E in the two-dimensional array DCC. S105-2, Recalculate the viewing position sequence corresponding to the average viewport trajectory of the j-th cluster; S105-3, The operation for the j-th cluster ends; S106. For j = 1, 2, ..., K, perform the following operations in sequence: S106-1. If the number of viewing port position vector samples contained in the j-th cluster is less than the threshold NC, proceed to step S106-2. Otherwise, construct a dataset using the viewing port position sequence corresponding to all viewing port position vector samples contained in the j-th cluster to complete the training of the viewing port position neural network prediction model corresponding to the j-th cluster. S106-2, The operation for the j-th cluster ends; 2) The second part of this method involves estimating the viewer type of the current viewer on a computer server, and selecting a corresponding neural network prediction model for the viewport position based on the estimated viewer type to predict the viewport position of the current frame. The specific steps are as follows: S201. Create a list LD containing K elements in the memory of the computer server. Each element of list LD stores a spherical distance value. Let the value of each element of list LD be INF2, where INF2 represents the maximum positive value that the elements of list LD can store. S202. For n1 = 2, 3, ..., N, perform the following operations in sequence: S202-1. Receive the actual viewing port position of the current viewer when watching the n1-1th frame of the panoramic video PanoV, which is transmitted from the client to the computer server, and save it in the memory of the computer server. S202-2. Arrange the actual viewing port positions of the current viewer when watching the panoramic video PanoV from the 1st to the n1-1th frames in ascending order according to their corresponding frame numbers to obtain a sequence containing n1-1 viewing port positions, which is called the viewing port position vector sample A01. S202-3. For j = 1, 2, ..., K, perform the following operations in sequence: S202-3-1. If the number of viewing port position vector samples contained in the j-th cluster is less than the threshold NC, proceed to step S202-3-3. Otherwise, the subsequence consisting of the 1st to the n1-1th viewing port positions of the viewing port position sequence corresponding to the average viewing port trajectory of the j-th cluster is called the viewing port position vector sample B_i. S202-3-2. Calculate the spherical distance D2 between the viewing port position vector sample A01 and the viewing port position vector sample B_i, D2 = d_1 + d_2 + ... + d_(n1-1), where d_t represents the spherical distance between the points corresponding to the t-th viewing port position of the viewing port position sequence corresponding to the viewing port position vector sample A01 and the viewing port position vector sample B_i on the panoramic imaging sphere, t = 1, 2, ..., (n1-1); assign the spherical distance D2 to the j-th element of the list LD; S202-3-3, The operation for the j-th cluster ends; S202-4. Find the index Ik of the smallest element in list LD; S202-5. Estimate the type of the current viewer as the viewer type corresponding to the Ikth cluster; S202-6. Using the neural network prediction model corresponding to the Ikth cluster, predict the viewing position of the current viewer when watching the n1st frame based on the actual viewing position of the current viewer when watching the panoramic video PanoV from the 1st to the n1-1th frames. S202-7, Operations on the n1st frame have ended.

Images