Panoramic video streaming transmission viewing viewport trajectory clustering analysis method

Abstract

The invention discloses a panoramic video streaming transmission viewing viewport trajectory clustering analysis method. The objective of the method is to perform clustering analysis on viewport viewing tracks of different viewers of the same panoramic video so as to obtain behavior characteristics of different types of viewers when watching the panoramic video. According to the method, the distance calculation between any two viewport positions on the spherical panoramic picture can be correctly processed, so that the average viewport track of each clustering cluster calculated according to each viewing viewport position vector sample clustering cluster obtained by clustering is ensured; and behavior characteristics of different types of viewers when watching the panoramic video can be correctly reflected. The method has the advantages of high flexibility and adaptability when clustering analysis is carried out on viewing viewport tracks of different panoramic videos, and the purpose of generating different numbers of viewing viewport position vector sample clusters for different panoramic videos can be conveniently achieved.

Description

Technical Field

[0001] This invention relates to a method for clustering and analyzing viewport trajectories during panoramic video streaming, belonging to the technical field of panoramic video-on-demand transmission and user viewing behavior feature analysis. Background Technology

[0002] In panoramic video-on-demand transmission and viewing applications, the viewing viewpoint position change trajectory of each frame during a viewer's viewing of a panoramic video can be recorded and stored as historical data on a computer server. Cluster analysis of the historical viewing viewpoint position change trajectories of a large number of viewers can reveal the characteristics of different types of viewers' viewing viewpoint position trajectory changes when watching the same panoramic stereoscopic video. This allows for the acquisition of behavioral characteristics of different types of viewers when watching panoramic videos, which can then be applied in practice. For example, when predicting the viewing viewpoint position of a new viewer during their viewing of a panoramic video, their user type can be estimated first based on the average trajectory of each cluster of the aforementioned viewing viewpoint position change trajectories and the viewer's viewing viewpoint position change trajectory before the current moment. Then, the viewing viewpoint position prediction algorithm can be adjusted based on the estimated user type to adaptively improve the accuracy of the prediction. Therefore, cluster analysis of the historical viewing viewpoint position change trajectories of different viewers of the same panoramic video has significant practical implications.

[0003] Equirectangular Projection (ERP) panoramic images use two-dimensional rectangular images to store panoramic spherical image signals of 360 degrees horizontally + 180 degrees vertically. Figure 1 In the ERP panoramic view, two viewing ports are shown, namely Viewport 1 and Viewport 2. Each viewing port actually corresponds to a viewing cone. For example... Figure 2 As shown, in this invention, the position where the ray emitted from the viewer's viewpoint, passing through the center point of the bottom surface of the viewing cone, intersects the panoramic imaging sphere (a sphere with radius 1 and center at the viewpoint) is called the position of the viewing port on the spherical panoramic image. Note: According to the rules of equidistant cylindrical projection, the position on the spherical panoramic image can also be projected onto the ERP panoramic image. Figure 1 The solid circle position in the image refers to the projection position on the ERP panoramic image where the ray emanating from the viewer's viewpoint intersects the center points of the bottom surfaces of the two view frustums corresponding to viewports 1 and 2 with the panoramic imaging sphere. It's worth noting that the aforementioned viewport position change trajectory is actually a sequence of viewport positions (a discrete position sequence) as the viewer watches each frame of the panoramic video, which is not the same as the continuous position trajectory of an object moving in three-dimensional space.

[0004] Section 9.6 of the 2016 Tsinghua University Press textbook *Machine Learning* introduces hierarchical clustering, which involves calculating the distance between clusters. Formula (9.42) in the textbook represents the maximum distance formula between clusters, which can be used to calculate the maximum distance between clusters. To calculate the maximum distance between clusters, it is necessary to be able to calculate the distance between any two samples. In clustering algorithms, Minkowski distance is a common distance metric, while Euclidean distance and Manhattan distance are two special cases of Minkowski distance. As mentioned earlier, if viewed on a panoramic sphere, the aforementioned viewport position corresponds to a point on the sphere. Since a panoramic image is essentially a spherical image, the distance between two different viewport positions should be the spherical distance between their corresponding points on the spherical panoramic image. Therefore, Minkowski distance is not suitable for calculating the distance between two sequences of viewport positions. This invention will solve this problem. For two viewing port position sequences (which can be considered as two viewing port position vector samples) of the same panoramic video, this invention defines the spherical distance D between them as: D = d_1 + d_2 + … + d_N; d_n represents the spherical distance between the points corresponding to the nth viewing port position of each of the two viewing port position sequences on the panoramic imaging sphere, n = 1, 2, …, N, where N represents the number of viewing port positions in the viewing port position sequence. In this invention, the spherical panoramic image is defined on the panoramic imaging sphere, the radius of the sphere corresponding to the panoramic imaging sphere is 1, and the viewing video frame numbers corresponding to the same position of the viewing port position in any two viewing port position sequences are the same.

[0005] The AGNES hierarchical clustering algorithm is introduced in Algorithm 9.11 of Section 9.6 of the book *Machine Learning*, published by Tsinghua University Press in 2016. Line 11 of the algorithm description states that if the current number of clusters q > k, the cluster merging operation continues, merging the two closest clusters (see lines 12-22 of Algorithm 9.11), where k represents the preset number of clusters. For panoramic video viewing port trajectory clustering, it is not easy to determine the specific value of k in advance in practice. Therefore, designing a reasonable stopping condition for the cluster merging operation is very meaningful. In this invention, the stopping condition for the cluster merging operation of the viewing port position sequence is designed as follows: if the maximum distance between the two closest clusters divided by N is greater than a given spherical distance threshold Td, then the cluster merging operation stops, and the AGNES hierarchical clustering operation is completed. Summary of the Invention

[0006] The purpose of this invention is to provide a method for clustering and analyzing the viewing port trajectory of panoramic video streaming. This method can perform clustering analysis on the viewing port trajectories of different viewers of the same panoramic video, and calculate the viewing port position sequence corresponding to the average viewing port trajectory of different clusters, thereby obtaining the behavioral characteristics of different types of viewers when watching panoramic videos.

[0007] This method is implemented as follows, including the following steps:

[0008] S1. Read all viewing port position sequences of the panoramic video PanoV from the disk storage of the computer server and save them in the memory of the computer server; each viewing port position sequence contains N viewing port positions; N is a positive integer; treat each viewing port position sequence as a viewing port position vector sample, and all viewing port position vector samples of the panoramic video PanoV constitute the viewing port position vector sample set DS.

[0009] S2. Perform hierarchical clustering on the viewport position vector sample set DS using the AGNES hierarchical clustering algorithm; use the maximum distance function dmax between clusters of the viewport position vector sample clusters as the distance metric function between clusters required by the AGNES hierarchical clustering algorithm; when performing the AGNES hierarchical clustering operation, the stopping condition for the cluster merging operation of the viewport position vector sample clusters is: if the distance between the two closest viewport position vector sample clusters divided by N is greater than the spherical distance threshold Td, then stop the cluster merging operation, and the hierarchical clustering operation is completed;

[0010] S3. Output the clustering results of the viewport position vector samples generated by the AGNES hierarchical clustering operation: {C_1, C_2, …, C_K}, where K represents the final number of clusters, C_i represents the i-th cluster, i = 1, 2, …, K; C_i is a set of viewport position vector samples.

[0011] S4. For i = 1, 2, …, K, perform the following operations respectively:

[0012] S4-1. For n = 1, 2, …, N, perform the following operations respectively:

[0013] ① Represent the positions of the nth viewport position in the viewing viewport position sequence corresponding to each viewing viewport position vector sample in C_i on the panoramic imaging sphere as Pn_1, Pn_2, ..., Pn_M, respectively, where M represents the number of viewing viewport position vector samples in C_i, and Pn_m is the position of the nth viewport position in the viewing viewport position sequence corresponding to the mth viewing viewport position vector sample in C_i on the panoramic imaging sphere, m = 1, 2, ..., M;

[0014] ② Calculate the average vector Vn = (Vn_1 + Vn_2 + … + Vn_M) / M, where Vn_m represents the vector from... Figure 2 The viewpoint shown is the unit vector pointing to the position Pn_m on the panoramic imaging sphere, where m = 1, 2, …, M;

[0015] ③ Calculate the normalized vector Vn1 of the average vector Vn;

[0016] ④ Calculate the intersection point PSEC of the ray originating from the viewpoint along the normalized vector Vn1 direction and the panoramic imaging sphere. Take the viewport position corresponding to the intersection point PSEC as the nth viewport position of the viewing viewport position sequence corresponding to the average viewport trajectory of the i-th cluster.

[0017] ⑤ The operation on the nth viewport position in the viewport position sequence corresponding to each viewport position vector sample in the i-th cluster ends;

[0018] S4-2, The operation on the i-th cluster ends.

[0019] In step S2, the maximum distance function dmax of the cluster of viewing port position vector samples is calculated using the maximum distance formula between clusters in AGNES hierarchical clustering. The distance between two viewing port position vector samples is measured using spherical distance D, where D = d_1 + d_2 + … + d_N, and d_n represents the spherical distance between the points corresponding to the nth viewing port position of each of the two viewing port position vector samples on the panoramic imaging sphere, n = 1, 2, …, N.

[0020] The positive effects of this invention are: this method can correctly handle the distance calculation between any two viewport positions on a spherical panoramic image, thereby ensuring that the average viewport trajectory of each cluster calculated based on the clusters of viewport position vector samples obtained through clustering can accurately reflect the behavioral characteristics of different types of viewers when watching panoramic videos. Furthermore, this method controls when to stop merging viewport position vector sample clusters in AGNES hierarchical clustering based on the spherical distance threshold Td. This provides better flexibility and adaptability when performing clustering analysis on the viewport trajectories of different panoramic videos, and can easily achieve the goal of generating different numbers of viewport position vector sample clusters for different panoramic videos. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the viewing port on the ERP panoramic screen.

[0022] Figure 2This is a schematic diagram showing the relationship between the viewing cone and the viewport position on the panoramic imaging sphere. Detailed Implementation

[0023] To make the features and advantages of this method clearer, the method is further described below with reference to specific embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. This embodiment considers panoramic video stored on a computer server. Viewers obtain panoramic video data via streaming over a network on a client and view the panoramic video using a head-mounted display. The client transmits the viewport position of the viewer in each frame of the panoramic video to the computer server in real time, and this information is stored on the computer server as historical data.

[0024] This method is implemented as follows, including the following steps:

[0025] S1. Read all viewing port position sequences of the panoramic video PanoV from the disk storage of the computer server and save them in the memory of the computer server; each viewing port position sequence contains N viewing port positions; N is a positive integer; treat each viewing port position sequence as a viewing port position vector sample, and all viewing port position vector samples of the panoramic video PanoV constitute the viewing port position vector sample set DS.

[0026] S2. Perform hierarchical clustering on the viewport position vector sample set DS using the AGNES hierarchical clustering algorithm; use the maximum distance function dmax between clusters of the viewport position vector sample clusters as the distance metric function between clusters required by the AGNES hierarchical clustering algorithm; when performing the AGNES hierarchical clustering operation, the stopping condition for the cluster merging operation of the viewport position vector sample clusters is: if the distance between the two closest viewport position vector sample clusters divided by N is greater than the spherical distance threshold Td, then stop the cluster merging operation, and the hierarchical clustering operation is completed;

[0027] S3. Output the clustering results of the viewport position vector samples generated by the AGNES hierarchical clustering operation: {C_1, C_2, …, C_K}, where K represents the final number of clusters, C_i represents the i-th cluster, i = 1, 2, …, K; C_i is a set of viewport position vector samples.

[0028] S4. For i = 1, 2, …, K, perform the following operations respectively:

[0029] S4-1. For n = 1, 2, …, N, perform the following operations respectively:

[0030] ① Represent the positions of the nth viewport position in the viewing viewport position sequence corresponding to each viewing viewport position vector sample in C_i on the panoramic imaging sphere as Pn_1, Pn_2, ..., Pn_M, respectively, where M represents the number of viewing viewport position vector samples in C_i, and Pn_m is the position of the nth viewport position in the viewing viewport position sequence corresponding to the mth viewing viewport position vector sample in C_i on the panoramic imaging sphere, m = 1, 2, ..., M;

[0031] ② Calculate the average vector Vn = (Vn_1 + Vn_2 + … + Vn_M) / M, where Vn_m represents the vector from... Figure 2 The viewpoint shown is the unit vector pointing to the position Pn_m on the panoramic imaging sphere, where m = 1, 2, …, M;

[0032] ③ Calculate the normalized vector Vn1 of the average vector Vn;

[0033] ④ Calculate the intersection point PSEC of the ray originating from the viewpoint along the normalized vector Vn1 direction and the panoramic imaging sphere. Take the viewport position corresponding to the intersection point PSEC as the nth viewport position of the viewing viewport position sequence corresponding to the average viewport trajectory of the i-th cluster.

[0034] ⑤ The operation on the nth viewport position in the viewport position sequence corresponding to each viewport position vector sample in the i-th cluster ends;

[0035] S4-2, The operation on the i-th cluster ends.

[0036] In step S2, the maximum distance function dmax of the cluster of viewing port position vector samples is calculated using the maximum distance formula between clusters in AGNES hierarchical clustering (see formula (9.42) in "Machine Learning" published by Tsinghua University Press in 2016). The distance between two viewing port position vector samples is measured using spherical distance D, where D = d_1 + d_2 + … + d_N, and d_n represents the spherical distance between the points corresponding to the nth viewport position of each of the two viewing port position vector samples on the panoramic imaging sphere, n = 1, 2, …, N.

[0037] In this embodiment, the spherical distance threshold Td = π / 10. Step S3 obtains the clustering results of the viewing port position vector sample set DS after clustering. Step S4 calculates the viewing port position sequence corresponding to the average viewing port trajectory of each cluster, which reflects the behavioral characteristics of the viewer type corresponding to each cluster. The direction of the normalized vector Vn1 is the same as the direction of the average vector Vn, and the length of the normalized vector Vn1 is 1.

Claims

1. A method for clustering and analyzing viewport trajectories during panoramic video streaming, characterized in that: The technical solution of this method is implemented as follows: Perform the following steps: S1. Read all viewing port position sequences of the panoramic video PanoV from the disk storage of the computer server and save them in the memory of the computer server; each viewing port position sequence contains N viewing port positions; N is a positive integer; treat each viewing port position sequence as a viewing port position vector sample, and all viewing port position vector samples of the panoramic video PanoV constitute the viewing port position vector sample set DS. S2. Perform hierarchical clustering on the viewport position vector sample set DS using the AGNES hierarchical clustering algorithm; use the maximum distance function dmax between clusters of the viewport position vector sample clusters as the distance metric function between clusters required by the AGNES hierarchical clustering algorithm; when performing the AGNES hierarchical clustering operation, the stopping condition for the cluster merging operation of the viewport position vector sample clusters is: if the distance between the two closest viewport position vector sample clusters divided by N is greater than the spherical distance threshold Td, then stop the cluster merging operation, and the hierarchical clustering operation is completed; S3. Output the clustering results of the viewport position vector samples generated by the AGNES hierarchical clustering operation: {C_1, C_2, …, C_K}, where K represents the final number of clusters, C_i represents the i-th cluster, i = 1, 2, …, K; C_i is a set of viewport position vector samples. S4. For i = 1, 2, …, K, perform the following operations respectively: S4-1. For n = 1, 2, …, N, perform the following operations respectively: ① Represent the positions of the nth viewport position in the viewing viewport position sequence corresponding to each viewing viewport position vector sample in C_i on the panoramic imaging sphere as Pn_1, Pn_2, ..., Pn_M, respectively, where M represents the number of viewing viewport position vector samples in C_i, and Pn_m is the position of the nth viewport position in the viewing viewport position sequence corresponding to the mth viewing viewport position vector sample in C_i on the panoramic imaging sphere, m = 1, 2, ..., M; ② Calculate the average vector Vn = (Vn_1 + Vn_2 + … + Vn_M) / M, where Vn_m represents the unit vector pointing from the viewpoint to the position Pn_m on the panoramic imaging sphere, m = 1, 2, …, M; ③ Calculate the normalized vector Vn1 of the average vector Vn; ④ Calculate the intersection point PSEC of the ray originating from the viewpoint along the normalized vector Vn1 direction and the panoramic imaging sphere. Take the viewport position corresponding to the intersection point PSEC as the nth viewport position of the viewing viewport position sequence corresponding to the average viewport trajectory of the i-th cluster. ⑤ The operation on the nth viewport position in the viewport position sequence corresponding to each viewport position vector sample in the i-th cluster ends; S4-2, The operation for the i-th cluster ends; In step S2, the maximum distance function dmax of the cluster of viewing port position vector samples is calculated using the maximum distance formula between clusters in AGNES hierarchical clustering. The distance between two viewing port position vector samples is measured using spherical distance D, where D = d_1 + d_2 + … + d_N, and d_n represents the spherical distance between the points corresponding to the nth viewing port position of each of the two viewing port position vector samples on the panoramic imaging sphere, n = 1, 2, …, N.

Images