Method and apparatus for multi-user view prediction and rate allocation

Abstract

The application discloses a kind of based on multi-user perspective prediction and code rate allocation method and device, wherein method includes: the history head trajectory of user is converted into viewpoint trajectory;Based on multi-user viewpoint trajectory, using clustering method obtains the ROI region of each frame of panorama video;Intermediate viewpoint is obtained by using target user history viewpoint trajectory training time sequence model and prediction;The viewpoint trajectory similarity of target user and other user is calculated, and prediction viewpoint trajectory weight model is constructed according to viewpoint trajectory similarity;Combine viewpoint trajectory weight, and intermediate viewpoint is adjusted to obtain final prediction viewpoint using other user viewpoint trajectory;The FOV region of current frame of panorama video is constructed using final prediction viewpoint;Code rate is allocated for current video frame in combination with FOV and ROI region.The application greatly reduces the demand for computing resources, makes full use of transmission bandwidth, while ensuring the transmission quality of video, optimizes user experience.

Description

Technical Field

[0001] The present invention belongs to the field of panoramic video, and relates to a panoramic video user perspective prediction and bit rate allocation strategy, and specifically to a perspective prediction and bit rate allocation device and working method based on multi-user historical head trajectories. Background Art

[0002] Panoramic video, also known as 360-degree video, is a three-dimensional video that allows users to change the viewing direction at will. The main drawback of traditional two-dimensional video is its lack of interactivity. Users can only passively watch the video content, unable to freely turn their heads to view different scenes as they would in the real world. The emergence of panoramic video has broken this deadlock. Currently, panoramic videos seen on major online platforms are mainly shot by special panoramic cameras (typically composed of several ordinary cameras that can capture images from multiple angles). These cameras then stitch together the video images from various angles into a 360-degree view through a series of steps including feature point detection, feature matching, and fusion. To watch panoramic videos, users need to wear special hardware such as VR glasses or a head-mounted display (HMD). The user then views the 360-degree video from the position of the panoramic camera. Users can freely turn their heads, and the video image changes accordingly, just like experiencing it in a real environment.

[0003] Compared to general two-dimensional videos, panoramic videos are closer to the user's eyes and therefore have more stringent requirements for video resolution. Even under the same single-eye resolution, due to the need to retain 360-degree image information, the data volume of panoramic videos is much higher than that of ordinary videos, generally 5 to 6 times that of the latter. At the same time, panoramic videos have relatively strict requirements on the transmission latency of the network system. The industry believes that the MTP (Motion To Photos Latency) of the transmission system, that is, the response time from turning the head to switching the video screen, must be controlled within 20ms. Otherwise, due to physiological reasons, users will feel dizzy and nauseous, resulting in a very poor viewing experience.

[0004] The high data volume and latency-sensitive nature of panoramic video places significant strain on existing networks, making it a challenging task to provide high-quality panoramic video services to multiple users. In real-world viewing scenarios, due to the peculiarities of the human eye, users can only see a portion of the panoramic video at any given moment, known as the FOV (Field of View). Transmitting all video clips at high quality would not only place unprecedented pressure on the network but also result in unnecessary resource waste. Transmitting only the FOV at high quality ensures a superior user experience while also reducing network transmission pressure.

[0005] Current panoramic video transmission schemes are mainly divided into two types: full transmission and adaptive transmission. Full transmission is similar to the traditional video transmission scheme, that is, the panoramic video is completely transmitted to the user. The mainstream streaming platform (YouTube) currently uses this transmission method. This transmission scheme can ensure that the user sees all the video content, but it places a heavy burden on the network and causes a lot of bandwidth waste. The adaptive transmission scheme is to transmit differentially according to the viewport area. The Tile-based method is considered to be an ideal scheme for transmitting panoramic video. In the Tile-based transmission method, the panoramic video frame is divided into MxN Tiles, each of which can be independently encoded and transmitted. The Tiles in the FOV area are transmitted at high quality, and the Tiles in the non-FOV area are transmitted at low quality or not transmitted at all. This not only ensures the user's viewing experience, but also greatly reduces the burden on the network. Due to the delay requirements of panoramic video, the adaptive transmission of Tiles in the FOV area requires higher FOV prediction.

[0006] The existing methods for FOV prediction mainly include two types: one is based on the user's historical head trajectory when watching the video, and the other is based on the saliency detection of the video content. The former is based on the user's historical head trajectory, and trains a regression model or a time series model to predict the future FOV area. Within 1s, it can obtain high accuracy, but when the prediction time exceeds 2s, the prediction accuracy drops significantly, which cannot meet the FOV prediction requirements of the Tile-based transmission. The latter is based on the saliency of the video content to represent the user's attention to different areas. Generally speaking, areas with high saliency are more likely to be watched by users. By training an LSTM network model based on the saliency of the video content, the viewing probability of each area of the video is obtained, and then the FOV area is obtained. To some extent, this can improve the accuracy of FOV prediction. Some studies combine the two methods to predict the FOV. By training a regression model or a time series model based on the head motion trajectory, an intermediate viewport is predicted, and the intermediate viewport is adjusted based on the saliency of the video to obtain the final FOV, which improves the overall prediction accuracy of the FOV. However, video saliency detection requires a lot of computational resources and time overhead, especially for spherical panoramic videos. Since there is no efficient 3D image-based saliency detection method, additional computational overhead is required to convert spherical videos to planar videos and then project the planar videos back to spherical videos. The present application uses multi-user head motion trajectory and clustering to predict the FOV, which not only improves the long-term prediction accuracy, but also reduces the computational resources required for saliency detection. SUMMARY

[0007] The purpose of the present application is to overcome the above technical deficiencies, provide a kind of based on the view angle prediction and code rate allocation device and working method of multi-user history head trajectory, can effectively solve the problem of low prediction accuracy of long time panoramic video FOV, reduce video transmission delay and network load, ensure video quality and user experience.

[0008] The technical scheme adopted by the present application to solve the technical problems is a method based on multi-user view angle prediction and code rate allocation, comprising the following steps:

[0009] Step 1, convert the user historical head trajectory into a viewpoint trajectory;

[0010] Step 2, based on the multi-user viewpoint trajectory, use clustering method to obtain the ROI region of each frame of panoramic video;

[0011] Step 3, use the target user historical viewpoint trajectory to train the time series model and predict the intermediate viewpoint;

[0012] Step 4, calculate the viewpoint trajectory similarity of the target user and other users, and construct a predicted viewpoint trajectory weight model according to the viewpoint trajectory similarity;

[0013] Step 5, combine the viewpoint trajectory weight, and adjust the intermediate viewpoint using the viewpoint trajectory of other users to obtain the final predicted viewpoint;

[0014] Step 6, use the final predicted viewpoint to construct the FOV region of the current frame of panoramic video;

[0015] Step 7, combine FOV and ROI region to allocate code rate for the current video frame.

[0016] Further, the user historical head trajectory in step 1 is represented by quaternion, and is converted into vector coordinates (x, y, z) of the viewpoint using the following formula:

[0017]

[0018] Where (qx, qy, qz, qw) is the user historical head motion trajectory represented by quaternion; and the vector coordinates are converted into longitude and latitude coordinates (lon, lat) using the following formula:

[0019]

[0020] The viewpoint trajectory is represented by the above longitude and latitude coordinates (lon, lat).

[0021] Further, in step 2, based on the viewpoint trajectory of multiple users, a DBSCAN clustering method based on density is used to analyze the clustering of the viewpoint of each video frame, and the viewpoint dense area is represented as the ROI region.

[0022] Further, in step 3, the time sequence about longitude and latitude is respectively constructed according to the video playing time sequence by using the longitude and latitude coordinates of the target user historical viewpoint, and the ARIMA model about longitude and latitude is respectively trained by using the time sequence, and the middle viewpoint of the future video frame is predicted;

[0023] When constructing time sequence data, since the panoramic video is spherical, the viewpoint longitude coordinates of adjacent frames will appear discontinuous, and the following formula needs to be used for adjustment:

[0024]

[0025] Wherein, lon' f represents the adjusted longitude of the current frame, lon f , lon f-1 respectively represent the longitude of the current frame and the previous frame, and width represents the maximum longitude value of the panoramic video, which is 360 degrees.

[0026] Further, in step 4, the similarity between the historical viewpoint trajectory of the target user and the historical viewpoint trajectory of other users is calculated by using the Euclidean distance, and the calculation formula is as follows:

[0027]

[0028] Wherein, S i represents the similarity of the viewpoint trajectory of user i and the target user, T represents the total number of frames in the past time period, represents the longitude and latitude coordinates of the viewpoint of user i in the video frame f, lat f , lon f respectively represent the longitude and latitude coordinates of the viewpoint of the target user in the video frame f.

[0029] Then, according to the similarity threshold, it is determined whether the trajectory is similar. When the viewpoint trajectories of the two users are in a quarter of the Tile within a certain time, it is considered that the viewpoint trajectories of the two users are similar, so the similarity threshold is defined as follows:

[0030]

[0031] Wherein, M and N respectively represent the number of horizontal and vertical slices of the panoramic video.

[0032] Further, after obtaining the similarity between the viewpoint trajectory of other users and the target user in step 4, the weight of the viewpoint trajectory of each user to the target user is set according to the similarity, and the prediction viewpoint trajectory weight model is obtained, and the calculation formula is as follows:

[0033]

[0034] wherein W i represents the viewpoint coordinate adjustment weight of the multi-target user of user i, S i , S j respectively represent the trajectory similarity of users i and j with the target user, and Q represents the total number of users.

[0035] Further, the viewpoint trajectory adjustment weight calculated by the trajectory similarity in step 5 is used to adjust the intermediate viewpoint predicted in step 3, and the calculation formula is as follows:

[0036]

[0037] wherein respectively represent the intermediate viewpoint longitude and latitude coordinates of the target user in frame f predicted by the time sequence model, respectively represent the viewpoint longitude and latitude coordinates of user i in video frame f, and Q represents the total number of users similar to the target user in viewpoint trajectory; the intermediate viewpoint obtained by adjusting the viewpoint predicted by the time sequence model using the viewpoint trajectory of multiple users is used to obtain the predicted viewpoint coordinates lon f , lat f of the target user in video frame f.

[0038] Further, in step 6, a rectangular region with a horizontal angle of 110 degrees and a vertical angle of 90 degrees is constructed with the viewpoint coordinates obtained in step 5 as the center, that is, the FOV region.

[0039] Further, in step 7, the position of the Tile is combined to assign a code rate to the current video frame, and the specific implementation is as follows:

[0040] Step 71, according to the position of each Tile, a saliency value is assigned, and the higher the saliency, the greater the probability of being watched: if the Tile is in the FOV and belongs to the ROI region at the same time, the saliency of the Tile is 3; if the Tile is in the FOV but does not belong to the ROI region, the saliency of the Tile is 2; and the saliency of the Tile outside the FOV region is 1.

[0041] Step 72, the probability of being watched is calculated according to the saliency of the Tile, and the calculation formula is as follows:

[0042]

[0043]

[0044] wherein α sum represents the total saliency of all Tiles of the current video frame, M and N represent the number of rows and columns of the video frame, and α i，j , p i,j respectively represent the saliency and the probability of being watched of the i-th row and j-th column Tile.

[0045] Step 73, Tile code rate allocation formula is as follows:

[0046] B i,j =B T p i,j

[0047] Wherein B T , B i,j Respectively, the total code rate of video frame and Tile (i, j) code rate.

[0048] The application also provides a kind of based on multi-user perspective prediction and code rate allocation device, comprising data processing module, viewpoint prediction module and code rate allocation module;

[0049] Data processing module is to the head motion trajectory of multi-user watching panoramic video is handled to obtain viewpoint coordinates and ROI area of each frame of video, it includes data preprocessing submodule and clustering submodule;

[0050] Data preprocessing submodule collects the head motion trajectory of different users watching video, converts it into the viewpoint coordinates of each frame when user watches video, clustering submodule uses clustering method to carry out clustering analysis to the viewpoint coordinates of different users of same video frame, and sets high-density area as the ROI area of video;

[0051] Viewpoint prediction module is the subsequent module of data processing module, which obtains future viewpoint coordinates according to the user historical viewpoint coordinates obtained by data processing module, and viewpoint prediction module contains time series prediction submodule and viewpoint adjustment submodule, time series prediction submodule is responsible for training time series model using target user historical viewpoint trajectory, and intermediate viewpoint is predicted;Viewpoint adjustment submodule is responsible for calculating the historical viewpoint trajectory similarity of target user and other users using trajectory similarity function, constructs weight model according to similarity, adjusts intermediate viewpoint by combining weight model and other user's viewpoint trajectory, and obtains final predicted viewpoint;

[0052] Code rate allocation module is responsible for dividing the FOV area of current video frame of panoramic video, and dividing code rate for each Tile thereof;First, according to the final predicted viewpoint coordinates obtained by viewpoint prediction module, combine the area limit of human eye watching, and construct rectangular area as FOV with viewpoint as center;Second, according to the position of current video frame Tile, judge whether it is in FOV, whether it is in ROI, calculate its viewing probability, and allocate video code rate to probability.

[0053] Compared with prior art, the application has the following advantages and beneficial effects:

[0054] The method is suitable for a panoramic video transmission process, and a viewpoint is predicted by combining a multi-user head motion trajectory with a timing model, which has better prediction accuracy than a single user head motion trajectory in a long time; a video ROI region is obtained in a multi-user viewpoint clustering manner, which greatly reduces the demand for computing resources compared with obtaining a video ROI region by using saliency; a FOV region is obtained by combining viewpoint prediction and ROI region, and a code rate is allocated, so that the transmission bandwidth is fully utilized while the video transmission quality is ensured, and the user experience is optimized.

[0055] The data processing module of the viewpoint prediction and code rate allocation device obtains the viewpoint coordinates of each user and the ROI region of the video by processing the historical head motion trajectories of the multiple users; then, the viewpoint prediction module constructs a trajectory similarity function and a weight model according to the viewpoint coordinates of each user obtained by the data processing module, and trains a timing model to predict the intermediate viewpoint coordinates of a certain number of future video frames according to the historical viewpoint coordinates of the target user, adjusts the predicted intermediate viewpoint by combining the weight model and the viewpoint coordinates of other users to obtain the final predicted viewpoint; finally, the code rate allocation module constructs the FOV region according to the viewpoint obtained by the viewpoint prediction module, calculates the probability of being watched according to the position of each Tile of the video frame, and allocates a corresponding code rate to the Tile according to the probability. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 The method flowchart of the embodiment of the method.

[0057] Figure 2 The device principle diagram of the embodiment of the application. DETAILED DESCRIPTION

[0058] The technical solutions of the application will be further described below with reference to the drawings and embodiments.

[0059] In order to make the purpose, technical solutions and advantages of the application clearer, the application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0060] As shown in Figure 1 The example of the application provides a method based on multi-user viewpoint prediction and code rate allocation, which comprises the following steps:

[0061] Step 1: The HMD records the historical head motion trajectory of the user in the form of a quaternion, and converts the historical head motion of the quaternion into three-dimensional viewpoint coordinates (x, y, z) in the Cartesian coordinate system by using the following formula:

[0062]

[0063] Further, the three-dimensional viewpoint coordinates are converted into longitude and latitude coordinates (lon, lat) using the following formula:

[0064]

[0065] The viewpoint trajectory is represented by the above longitude and latitude coordinates (lon, lat).

[0066] Step 2: According to the viewpoint coordinates of multiple users in the same video frame, a density-based DBSCAN clustering method is used for clustering, and a high-density area is set as the ROI area of the video; the Eps and MinPts parameters of clustering need to be set and adjusted according to the actual clustering situation, and when the distance between the viewpoints of two different users does not exceed Eps, it can be considered that they belong to the same cluster and are interested in the same video area; the MinPts parameter can be set by referring to the total number of users, and generally more than one-third of the total number of users can be considered as a core point;

[0067] Step 3: The longitude and latitude time series data are constructed based on the historical viewpoint trajectory of the target user, and an ARIMA model is trained to predict the future viewpoint coordinates of the target user, i.e., the intermediate viewpoint; when constructing the time series data, since the panoramic video is spherical, the longitude coordinates of adjacent frames may be discontinuous, and the following formula needs to be used for adjustment:

[0068]

[0069] wherein lon f , lon f-1 represent the longitude of the current frame and the previous frame, respectively, and width represents the maximum longitude of the panoramic video, which is 360 degrees.

[0070] Step 4: If the viewpoint trajectories of different users have similarity in the past period of time, it is considered that their viewpoint trajectories also have similarity in the future period of time. Based on the above principle, a method is proposed to adjust the intermediate viewpoint predicted by the multi-time series model of other user viewpoint trajectories to obtain the final predicted viewpoint; wherein the Euclidean distance between viewpoints is used to calculate the similarity of user viewpoint trajectories, and the formula is as follows:

[0071]

[0072] wherein T represents the total number of frames of the video in the past period of time, represent the longitude and latitude coordinates of user i in video frame f, respectively, and lon f , lat f represent the longitude and latitude coordinates of the target user in video frame f.

[0073] Further, the step 4 calculates the viewpoint trajectory similarity of each user and the target user, which needs to be judged by the similarity threshold value whether the trajectory is similar; generally, when the viewpoint trajectories of two users are in a quarter of the Tile within a certain time, it is considered that the viewpoint trajectories of the two users are similar, so the similarity threshold value is defined as follows:

[0074]

[0075] Wherein, M, N are respectively the slice number of the panoramic video in horizontal and vertical directions; the similarity threshold value is not a fixed value, which can be adjusted according to the viewpoint prediction result in the actual prediction process, so as to achieve better prediction effect.

[0076] Further, the step 4 constructs a weight model according to the trajectory similarity of each user and the target user; it can be known from the similarity calculation formula that the smaller the similarity value is, the higher the similarity with the target user trajectory is, and when the similarity value tends to 0, it means that the two viewpoint trajectories are coincident; therefore, the weight model is constructed based on the trajectory similarity value, which is used to distribute the weight of each similar user in adjusting the intermediate viewpoint, and the weight model is as follows:

[0077]

[0078] Wherein, Q represents the total number of users similar to the target user viewpoint trajectory, S i , S j respectively represent the viewpoint trajectory similarity of user i, j and the target user.

[0079] Step 5, using the weight model, combining the viewpoint trajectory of each similar user to adjust the intermediate viewpoint predicted by the time sequence model to obtain the final predicted viewpoint, and the calculation formula is as follows:

[0080]

[0081] Wherein respectively represent the longitude and latitude coordinates of the intermediate viewpoint of the target user in frame f predicted by the time sequence model, respectively represent the longitude and latitude coordinates of the viewpoint of similar user i in video frame f, and Q represents the total number of users similar to the target user viewpoint trajectory; the intermediate viewpoint is adjusted by using the viewpoint trajectories of all similar users to obtain the predicted viewpoint coordinates lon f , lat f of the target user in video frame f.

[0082] Step 6, constructing FOV region according to final predicted viewpoint coordinates and assigning video code rate; constructing FOV region with 110 degrees of horizontal width and 90 degrees of vertical width as the center of the viewpoint; assigning code rate to each Tile of the video frame in combination with the position of the Tile; in general, FOV region and ROI region have a higher probability of being watched by users; therefore, if the Tile is in both FOV region and ROI region, it means that the Tile has the highest probability of being watched and is assigned the highest code rate, if the Tile is only in FOV region, it means that the Tile has a lower probability of being watched and is assigned a lower code rate, and if the Tile is not in FOV region, it means that the Tile has the lowest probability of being watched and is assigned the lowest code rate;

[0083] The specific manner is as follows:

[0084] ①According to the position of each Tile, assign a saliency value, the higher the saliency, the greater the probability of being watched by users: if the Tile is in FOV and also belongs to ROI region, the saliency of the Tile is 3; if the Tile is in FOV but does not belong to ROI region, the saliency of the Tile is 2; and the saliency of the Tile outside FOV region is 1;

[0085] ②Calculate the probability of being watched according to the saliency of the Tile, and the calculation formula is as follows:

[0086]

[0087]

[0088] wherein α sum represents the total saliency of all Tiles of the current video frame, M and N represent the number of rows and columns of the video frame divided, and α i，j , p i,j represent the saliency and the probability of being watched of the Tile in the i-th row and the j-th column, respectively.

[0089] ③The code rate assignment formula of the Tile is as follows:

[0090] B i,j = B T p i,j

[0091] wherein B T and B i,j represent the total code rate of the video frame and the code rate of the Tile (i, j), respectively.

[0092] The object of the viewpoint prediction in the method is a video frame, that is, the viewpoint coordinates are predicted once every interval video frame time, the time sequence model is trained according to the past viewpoint coordinates of the target user every frame, the intermediate viewpoint coordinates in a certain number of future video frames are predicted, the weight model is constructed by using the similarity of the viewpoint trajectories of other users and the target user, the intermediate viewpoint predicted by the time sequence model is adjusted by using the viewpoint coordinates of the same video frame of the trajectory similar user and the weight model, and the final prediction result is obtained.

[0093] The embodiment of the application provides a device based on multi-user viewpoint prediction and code rate allocation, which comprises a data processing module, a viewpoint prediction module and a code rate allocation module.

[0094] The data processing module is used for processing the head motion trajectory of multi-user watching panoramic video to obtain the viewpoint coordinates and the ROI region of each frame of video, and comprises a data preprocessing submodule and a clustering submodule; the data preprocessing submodule collects the head motion trajectory of different users watching video, and converts the head motion trajectory into the viewpoint coordinates of each frame of video watched by the user; the clustering submodule adopts a DBSCAN clustering mode based on density to perform clustering analysis on the viewpoint coordinates of different users of the same video frame, and sets a high-density region as the ROI region of the video.

[0095] The viewpoint prediction module is a subsequent module of the data processing module, and is used for predicting the future viewpoint coordinates according to the historical viewpoint coordinates of the user obtained by the data processing module. The viewpoint prediction module comprises a time sequence prediction submodule and a viewpoint adjustment submodule; the time sequence prediction submodule is responsible for training a time sequence model by using the historical viewpoint trajectory of the target user, and predicting an intermediate viewpoint; the viewpoint adjustment submodule is responsible for calculating the similarity of the historical viewpoint trajectory of the target user and other users by using a trajectory similarity function, constructing a weight model according to the similarity, and adjusting the intermediate viewpoint by combining the weight model and the viewpoint trajectory of other users to obtain a final predicted viewpoint.

[0096] The code rate allocation module is responsible for dividing the FOV region of the current video frame of the panoramic video, and allocating a code rate for each Tile. First, a rectangular region with the viewpoint as the center, a horizontal 110-degree and a vertical 90-degree is constructed as the FOV according to the viewpoint coordinates obtained by the viewpoint prediction module and in combination with the region limitation of the human eye watching; second, whether the current video frame Tile is located in the FOV or in the ROI is judged according to the position of the current video frame Tile, and the probability of being watched is calculated to allocate a video code rate for the Tile according to the probability.

[0097] The specific implementation of each module corresponds to each step, and the application will not be described.

[0098] It should be understood that parts not described in detail in the specification are all prior art.

[0099] The specific embodiments described herein are merely illustrative of the spirit of the application. Various modifications or changes in the specific embodiments described herein can occur to those skilled in the art to which the application pertains without departing from the spirit of the application, and it is understood that such modifications or changes are to be considered as within the scope of the application as defined by the appended claims.

Claims

1. A method based on multi-user perspective prediction and bitrate allocation, characterized in that: The steps include: Step 1: Convert the user's historical head trajectory into a viewpoint trajectory; Step 2: Based on the multi-user viewpoint trajectories, a clustering method is used to obtain the ROI area of ​​each frame of the panoramic video; Step 3: Use the target user's historical viewpoint trajectory to train a time series model and predict the intermediate viewpoints; Step 4: Calculate the viewpoint trajectory similarity between the target user and other users, and build a predicted viewpoint trajectory weight model based on the viewpoint trajectory similarity; Step 5: Combine the viewpoint trajectory weights and use the viewpoint trajectories of other users to adjust the intermediate viewpoint to obtain the final predicted viewpoint; Step 6: Use the final predicted viewpoint to construct the FOV area of ​​the current frame of the panoramic video; Step 7: Assign a bit rate to the current video frame based on the FOV and ROI area; In step 7, the bit rate is allocated to the current video frame based on the position of the tile. The specific implementation method is as follows; Step 71: Assign a saliency value to each tile based on its location. The higher the saliency, the greater the probability of being viewed by the user. If the tile is in the FOV and also in the ROI, the saliency of the tile is 3. If the tile is in the FOV but not in the ROI, the saliency is 2. The saliency of a tile outside the FOV is 1. Step 72: Calculate the probability of being viewed based on the tile significance. The calculation formula is as follows: where a sum Represents the sum of the saliency of all tiles in the current video frame, M and N represent the number of rows and columns into which the video frame is divided, and α i，j 、p i,j Represents the significance and viewing probability of the tile in row i and column j respectively; Step 73: The tile rate allocation formula is as follows: B i,j =B T p i,j Among them B T 、B i,j Represent the total bit rate of the video frame and the bit rate of Tile(i,j) respectively.

2. The method based on multi-user perspective prediction and bit rate allocation according to claim 1, wherein: The user's historical head trajectory in step 1 is represented by a quaternion and converted into the vector coordinates (x, y, z) of the viewpoint using the following formula: Where (qx, qy, qz, qw) is the user's historical head motion trajectory expressed as a quaternion; the vector coordinates are then converted into longitude and latitude coordinates (lon, lat) using the following formula: The viewpoint trajectory is represented by the above-mentioned longitude and latitude coordinates (lon, lat).

3. The method based on multi-user perspective prediction and bit rate allocation according to claim 1, wherein: In step 2, based on the viewpoint trajectories of multiple users, the density-based DBSCAN clustering method is used to perform cluster analysis on the viewpoints of each video frame, and the viewpoint-dense area is represented as the ROI area.

4. The method based on multi-user perspective prediction and bit rate allocation according to claim 1, wherein: In step 3, the target user's historical viewpoint latitude and longitude coordinates are used to construct a time series of latitude and longitude in the order of video playback time. The time series is used to train the ARIMA model for latitude and longitude, and the intermediate viewpoint of the future video frame is predicted. When constructing time series data, since panoramic videos are spherical, the viewpoint longitude coordinates of adjacent frames may be discontinuous. This needs to be adjusted using the following formula: Among them, lon' f Indicates the adjusted viewpoint longitude of the current frame, lon f ,lon f-1 They represent the viewpoint longitude of the current frame and the previous frame respectively. Witdth represents the maximum longitude of the panoramic video, which is 360 degrees.

5. The method based on multi-user view prediction and bit rate allocation according to claim 1, wherein: In step 4, the Euclidean distance is used to calculate the similarity between the historical viewpoint trajectories of other users and the historical viewpoint trajectories of the target user. The calculation formula is as follows: Among them S i represents the similarity between the viewpoint trajectory of user i and the target user, T represents the total number of video frames in the past time period, They represent the latitude and longitude coordinates of the viewpoint of user i in video frame f, lat f ,lon f Respectively represent the latitude and longitude coordinates of the target user’s viewpoint in the video frame f; Then, the similarity threshold is used to determine whether the trajectories are similar. When the viewpoint trajectories of two users are within a quarter of a tile within a certain period of time, the viewpoint trajectories of the two users are considered to be similar. The similarity threshold is defined as follows: Among them, M and N are the number of tiles in the horizontal and vertical directions of the panoramic video respectively.

6. The method based on multi-user perspective prediction and bit rate allocation according to claim 1, wherein: After obtaining the similarity between the viewpoint trajectories of other users and the target user in step 4, the weight of each user's viewpoint trajectory adjustment for the target user is set according to the similarity to obtain the predicted viewpoint trajectory weight model, which is calculated as follows: Where W i represents the viewpoint coordinate adjustment weight of multiple target users i, S i 、S j They represent the trajectory similarity between user i, j and the target user, and Q is the total number of users.

7. The method based on multi-user view prediction and bit rate allocation according to claim 1, wherein: In step 5, the viewpoint trajectory adjustment weight obtained by trajectory similarity calculation is used to adjust the intermediate viewpoint predicted in step 3. The calculation formula is as follows: in They represent the latitude and longitude coordinates of the intermediate viewpoint of the target user in frame f predicted by the time series model, Represent the latitude and longitude coordinates of the viewpoint of user i in video frame f, Q represents the total number of users with similar viewpoint trajectories to the target user; the intermediate viewpoints predicted by the time series model are adjusted using the viewpoint trajectories of multiple users to obtain the predicted viewpoint coordinates lon of the target user in video frame f. f ,lat f .

8. The method based on multi-user view prediction and bit rate allocation according to claim 1, wherein: In step 6, a rectangular area with a horizontal angle of 110 degrees and a vertical angle of 90 degrees is constructed with the viewpoint coordinates obtained in step 5 as the center, which is the FOV area.

9. An apparatus based on multi-user perspective prediction and bit rate allocation, for implementing the method based on multi-user perspective prediction and bit rate allocation according to any one of claims 1 to 8, characterized in that: Contains data processing module, viewpoint prediction module and bit rate allocation module; The data processing module processes the head motion trajectories of multiple users watching panoramic videos to obtain viewpoint coordinates and ROI areas of each video frame. It includes a data preprocessing submodule and a clustering submodule. The data preprocessing submodule collects head motion trajectories of different users watching videos and converts them into the viewpoint coordinates of each frame when the user watches the video. The clustering submodule uses a clustering method to perform cluster analysis on the viewpoint coordinates of different users in the same video frame and sets the high-density area as the video's ROI area; The viewpoint prediction module is a subsequent module of the data processing module. It predicts the future viewpoint coordinates based on the user's historical viewpoint coordinates obtained by the data processing module. The viewpoint prediction module includes a time series prediction submodule and a viewpoint adjustment submodule. The time series prediction submodule is responsible for training the time series model using the target user's historical viewpoint trajectory and predicting the intermediate viewpoints. The viewpoint adjustment submodule is responsible for calculating the similarity between the historical viewpoint trajectories of the target user and other users using the trajectory similarity function, building a weight model based on the similarity, and adjusting the intermediate viewpoints by combining the weight model and the viewpoint trajectories of other users to obtain the final predicted viewpoint. The bitrate allocation module is responsible for dividing the FOV area of ​​the current video frame of the panoramic video and allocating the bitrate for each of its tiles. First, based on the final predicted viewpoint coordinates obtained by the viewpoint prediction module and the area restrictions of human vision, a rectangular area centered on the viewpoint is constructed as the FOV. Second, based on the position of the current video frame tile, it determines whether it is within the FOV or within the ROI, calculates the probability of it being viewed, and allocates the video bitrate based on the probability.

Images