Fov viewpoint prediction method for transformer-based 3D point cloud video streaming transmission

Abstract

The application discloses a fov viewpoint prediction method for 3D point cloud video streaming based on a Transformer, and comprises the following steps: 1, constructing a 3D point cloud video saliency detection model based on a Transformer, and extracting saliency features of the 3D point cloud video by using the saliency detection model; 2, constructing a 3D point cloud video dynamic detection model based on user fov viewpoint prediction, and extracting fov viewpoint prediction features of the user by using the dynamic detection model; 3, finally, fusing the saliency features of the point cloud video and the dynamic features of the point cloud video to obtain the final predicted fov viewpoint of the user. The fov viewpoint prediction method for 3D point cloud video streaming based on the Transformer is first proposed, and the saliency detection and the dynamic fov viewpoint prediction are combined to obtain the final predicted fov viewpoint, so that a higher prediction accuracy can be realized.

Description

Technical Field

[0001] This invention belongs to the field of multimedia video transmission, specifically a fove viewpoint prediction method for 3D point cloud video streaming transmission based on Transformer. Background Technology

[0002] Point clouds are data structures that describe three-dimensional shapes, containing data such as the shape, color, and reflection intensity of three-dimensional objects. Therefore, compared to two-dimensional video, 3D point cloud video stores more dimensional information, resulting in a higher data volume than ordinary video. Since users can only see a small portion of each frame at any given time, each frame's point cloud contains a large number of redundant points. Transmitting the entire point cloud for each frame directly would consume significant memory and bandwidth. Therefore, it's necessary to predict the user's future FOV (Field of View) and transmit video based on the corresponding 3D point cloud region of interest to the user. This saves memory and bandwidth during video transmission while ensuring the quality and user experience of the viewed video.

[0003] In recent years, deep learning has been widely developed and applied in academia and industry. Viewpoint prediction methods combined with deep learning have become a hot research topic in the field of panoramic video. Some researchers have used LSTM networks for viewpoint prediction; however, the accuracy of this method drops significantly after 1 second, indicating low long-term prediction accuracy. Therefore, improving long-term prediction accuracy is a pressing issue for those skilled in the art. The Transformer model is naturally adept at handling unordered data, and its self-attention mechanism fully considers the correlation between points in different local areas. Therefore, using the Transformer model to predict the user's viewpoint is essential.

[0004] A method for classifying 3D point clouds based on limited training data (publication number: CN115690362A) published by Liang Guoqiang et al. of Guizhou University involves establishing a 3D point cloud model of the target point and randomly acquiring multiple 3D point cloud sample data. A Siamese network deep learning model is used for model training. The similarity and dissimilarity of features between sample data are learned to measure the similarity between two input sample data. Local neighborhood centroids are selected from the original point through farthest point sampling, and then local features are extracted using a PointNet network. The original point cloud data is sorted in ascending order along the X-axis, and several points are selected uniformly after setting a step size. The distance from each point to the origin is calculated using the Euclidean distance formula, and the distance feature is added to the PointNet network for global feature extraction. An improved PointNet network is used as a shared network of the Siamese network to establish a 3D point cloud classification model. All 3D point cloud data within the target point are collected, and the 3D point cloud classification model is used to classify the obtained 3D point cloud data. However, this method can only combine the spatial features of a single frame of point cloud video for 3D point cloud classification and does not consider the temporal features of multiple consecutive frames of point cloud video.

[0005] Dong Pingping et al. from Hunan Normal University disclosed a viewpoint prediction method, device, and medium for panoramic video (publication number: CN115086645A). The method involves acquiring the historical trajectory of the viewpoint of a target viewer while watching a panoramic video and converting it into a heatmap. The heatmap of the target user's historical viewpoint trajectory is then input into a seq2seq model to generate preliminary prediction results. Similar users matching the historical trajectory of the target user's viewpoint are found in expert data to obtain future information about these similar users. The preliminary prediction results and the future information of similar users are then input into an SE-Unet network model to output the final prediction result of the user's viewpoint. However, this method can only achieve high-precision viewpoint prediction for panoramic video, while its accuracy is low for point cloud video. Summary of the Invention

[0006] To address the shortcomings of the existing technology, this invention provides a Transformer-based fov viewpoint prediction method for 3D point cloud video streaming transmission, aiming to combine salient features and dynamic features to obtain the final predicted fov viewpoint, thereby improving prediction accuracy.

[0007] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0008] The present invention provides a method for predicting the viewpoint of 3D point cloud video streaming media based on Transformer, characterized by the following steps:

[0009] Step 1: Collection and preprocessing of viewpoint data:

[0010] Step 1.1: Collect and extract a×b historical FOV viewpoint information from a users watching b consecutive frames of point cloud video. in, This represents the FOV viewpoint information when the a-th user views the b-th frame of the point cloud video;

[0011] Step 1.2: Extract the a×b historical FOV viewpoint information. b-frame point cloud data (x1, x2, ..., x) from the user's perspective t ,…,x b ), where x t Let t represent the fovo viewpoint information of a users watching the t-th frame of the point cloud video, where t∈(1,b);

[0012] Step 2: Construct a viewpoint prediction model for 3D point cloud video streaming transmission based on Transformer, including: a 3D point cloud video saliency detection model based on Transformer, a 3D point cloud video dynamics detection model based on foveo viewpoint prediction, a saliency feature and dynamic feature fusion module, and a decoding network.

[0013] Step 2.1: The Transformer-based 3D point cloud video saliency detection model includes: a Transformer-based spatial feature extraction module M. TRS and Transformer-based temporal feature extraction module M TRT ;

[0014] Among them, the spatial feature extraction module M TRS Includes: Single-frame point embedding module M PE and the single-frame Transformer spatial feature extraction module M TFBS The time feature extraction module M TRT Includes: Multi-frame point embedding module M PED Multi-frame Transformer spatial feature extraction module M TFBDS and the multi-frame Transformer temporal feature extraction module M TFBDT ;

[0015] The spatial feature extraction module M TRS Using the 3D point cloud data x of frame t t As input, and passed through the single-frame point embedding module M PE The point cloud core points are processed by convolution to aggregate the 3D point cloud data x of the t-th frame. t The local geometry and contextual information are used to obtain the aggregated local neighborhood features F of frame t. t ;

[0016] The aggregated local neighborhood features F t Input the single-frame Transformer spatial feature extraction module M TFBS After feature extraction, the spatial scale features of frame t are obtained.

[0017] The time feature extraction module M TRT Using consecutive frames of 3D point cloud data (x) from frame t-τ to frame t-1 t-τ ,…,x t-2 ,x t-1 The input is used as the multi-frame point embedding module M. PED Convolutional processing of core points in point clouds to aggregate consecutive multi-frame 3D point cloud data (x t-τ ,…,x t-2 ,x t-1 The local geometry and contextual information of the region are used to obtain the local neighborhood features (F) aggregated from frame t-τ to frame t-1. t-τ ,…,F t-2 ,F t-1 );

[0018] The local neighborhood features (F) of consecutive frames aggregated from frame t-τ to frame t-1 t-τ ,…,F t-2 ,F t-1 Input the multi-frame Transformer spatial feature extraction module M TFBDS After feature extraction, the spatial scale features from frame t-τ to frame t-1 are obtained.

[0019] Spatial scale features from frame t-τ to frame t-1 Spatial scale features of frame t The multi-frame Transformer temporal feature extraction module M is input together. TFBDT The time-scale features of frame t are obtained through processing.

[0020] Step 2.2: Concatenate the spatial scale features of the t-th frame by channel. and the temporal scale features of the t-th frame Obtain the spatiotemporal features of frame t The spatiotemporal features That is, the 3D point cloud data x of the t-th frame in the 3D point cloud video. t The salient features;

[0021] Step 2.3: The 3D point cloud video dynamics detection model based on fov viewpoint prediction includes: an LSTM module and a fov viewpoint embedding module M. PEF and Transformer viewpoint feature extraction module M TFBF ;

[0022] The 3D point cloud video dynamic detection model uses the historical FOV viewpoint information. As input, and after viewpoint prediction by the LSTM module, a×b FOV viewpoint prediction information is obtained when a users watch b frames of point cloud video. in, This represents the FOV viewpoint prediction information when user a watches the point cloud video of frame b.

[0023] Prediction information from the a×b FOV viewpoints The FOV viewpoint information of the m-th user watching the t-th frame of the point cloud video is randomly selected. Then, the fov viewpoint information is extracted. Point cloud data x′ m,t , where m∈(1,a);

[0024] The point cloud data x′ m,t Input the fov viewpoint embedding module M PEF Perform convolution on the core points of the point cloud to aggregate the point cloud data x′. m,t Local geometry and contextual information are used to obtain x′ m,t Aggregated local neighborhood features F m,t ;

[0025] The aggregated local neighborhood features F m,t Input the Transformer viewpoint feature extraction module M TFBF Feature extraction is performed to obtain the predicted point cloud features of the viewpoint when the m-th user watches the point cloud video of frame t. And serve as a dynamic feature when the m-th user views the t-th frame of the point cloud video;

[0026] Step 2.4, the saliency features of the point cloud video in frame t. Dynamic features of the point cloud video of the m-th user at frame t After fusion processing in the input saliency feature and dynamic feature fusion model, the total feature F′ of the m-th user in the t-th frame is obtained. m,t ;

[0027] Step 2.5: The decoding network uses the total features F′ of the m-th user in the t-th frame. m,tAs input, after passing through the decoding network, the final predicted FOV viewpoint image of the m-th user in the t-th frame is obtained;

[0028] Step 3: Model Training

[0029] Based on b-frame point cloud data (x1, x2, ..., x t ,…,x b The gradient descent method is used to train the fov viewpoint prediction model for 3D point cloud video streaming transmission based on Transformer, and the Diou loss is used as the loss function to adjust the parameters of the model. When the number of training iterations reaches the set number or the loss function tends to a stable value, the training stops, thereby obtaining the optimal fov viewpoint prediction model, which is used to predict the user's fov viewpoint.

[0030] The FOV viewpoint prediction method for 3D point cloud video streaming transmission based on Transformer described in this invention is also characterized by the single-frame Transformer spatial feature extraction module M in step two. TFBS It consists of two feature extraction units (FEU) and two upsampling modules (M). US The structure of each feature extraction unit (FEU) consists of: a downsampling module M. DS and offset attention and residual module M OAR And extract spatial scale features according to the following steps.

[0031] 1) The downsampling module M in the first feature extraction unit (FEU) DS The aggregated local neighborhood features F t Perform farthest point sampling to obtain sampling point features.

[0032] Use the KNN algorithm to analyze the features of the sampling points. A query is performed to obtain the grouping index, and the grouping index is aggregated and projected before Max Pooling is applied to obtain the downsampling module M. DS Output downsampled features

[0033] 2) The downsampling features Input the offset attention and residual module M OAR After feature extraction, we obtain the effective aggregated long-range contextual features F′. t And the features output by the first feature extraction unit FEU;

[0034] 3) The context feature F′ t Input the first upsampling module MUS In the middle, after projection through the LN linear layer, interpolation is performed to obtain the first-level features after segmentation.

[0035] 4) The context feature F′ t After being processed by the second feature extraction unit (FEU), the context features F″ output by the second feature extraction unit are obtained. t ;

[0036] 5) The context feature F″ t Input the second upsampling module M US After processing, the segmented second-level features are obtained.

[0037] 6) The first-level features after segmentation Second-level features Connecting along dimensions yields spatial scale features with multi-level characteristics.

[0038] The fusion model of salient features and dynamic features in step 2.4 is performed according to the following steps:

[0039] 1) Utilize Fourier transform to analyze the saliency features of the point cloud video in frame t. Dynamic features of the point cloud video in frame t After averaging, the initial total feature F”, which reflects the combined level of dynamic and significant characteristics, is obtained. m ′ ,t ;

[0040] 2) The initial total feature F” m ′ ,t Perform region aggregation processing to obtain the region aggregation feature F” m,t ;

[0041] 3) Aggregate the region features F” m,t Normalization is performed to obtain the normalized region aggregation features. Then, serialization is performed to obtain the total feature F′ of frame t. m,t .

[0042] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing any of the FOV viewpoint prediction methods, and the processor is configured to execute the program stored in the memory.

[0043] The present invention provides a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, performs any of the steps of the FOV viewpoint prediction method.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] 1. This invention proposes for the first time a point cloud video saliency detection method based on Transformer. This method utilizes a point embedding module to aggregate the local neighborhood features of each point in the point cloud video, and at the same time uses a Transformer feature extraction module to extract the spatiotemporal features of the point cloud video, thereby improving the accuracy of saliency feature detection in 3D point cloud video.

[0046] 2. This invention proposes for the first time a 3D point cloud video fov viewpoint prediction method based on Transformer, which combines saliency detection with dynamic fov viewpoint prediction to obtain the final predicted fov viewpoint, thereby improving the accuracy of viewpoint prediction. Attached Figure Description

[0047] Figure 1 This is a system block diagram of the FOV viewpoint prediction method for 3D point cloud video streaming transmission based on Transformer proposed in this invention.

[0048] Figure 2 This is a block diagram of the 3D point cloud video saliency detection system based on Transformer proposed in this invention;

[0049] Figure 3 The system block diagram of the proposed single-frame Transformer spatial feature extraction model is shown in this invention. Detailed Implementation

[0050] In this embodiment, a Transformer-based 3D point cloud video streaming media transmission foveo viewpoint prediction method is described, such as... Figure 1 The diagram shows a complete system architecture for a Transformer-based 3D point cloud video streaming media transmission foveo (FOV) viewpoint prediction method. This system combines the saliency features of the point cloud video with the dynamic features of the user's FOV viewpoint to predict the user's FOV viewpoint when watching a particular frame of point cloud video. Figure 1 As shown, the method mainly includes the following steps:

[0051] Step 1: Collection and preprocessing of viewpoint data:

[0052] Step 1.1: Collect and extract a×b historical FOV viewpoint information from a users watching b consecutive frames of point cloud video. in, This represents the FOV viewpoint information when the a-th user views the b-th frame of the point cloud video;

[0053] Step 1.2: Extract a×b historical FOV viewpoint information. b-frame point cloud data (x1, x2, ..., x) from the user's perspective t ,…,x b ), where x t Let t represent the fovo viewpoint information of a users watching the t-th frame of the point cloud video, where t∈(1,b);

[0054] Step 2: Construct a viewpoint prediction model for 3D point cloud video streaming transmission based on Transformer, including: a 3D point cloud video saliency detection model based on Transformer, a 3D point cloud video dynamics detection model based on foveo viewpoint prediction, a saliency feature and dynamic feature fusion module, and a decoding network.

[0055] Step 2.1: A 3D point cloud video saliency detection model based on Transformer, such as... Figure 2 As shown, it includes: a Transformer-based spatial feature extraction module M. TRS and Transformer-based temporal feature extraction module M TRT ;

[0056] Among them, the spatial feature extraction module M TRS Includes: Single-frame point embedding module M PE and the single-frame Transformer spatial feature extraction module M TFBS Time Feature Extraction Module M TRT Includes: Multi-frame point embedding module M PED Multi-frame Transformer spatial feature extraction module M TFBDS and the multi-frame Transformer temporal feature extraction module M TFBDT ;

[0057] Spatial Feature Extraction Module M TRS Using the 3D point cloud data x of frame t t As input, and passed through the single-frame point embedding module M PE Convolutional processing of the core points of the point cloud to aggregate the 3D point cloud data x of frame t. t The local geometry and contextual information are used to obtain the aggregated local neighborhood features F of frame t. t ;

[0058] Aggregated local neighborhood features F t Input single-frame Transformer spatial feature extraction module MTFBS Feature extraction is performed during the single-frame Transformer spatial feature extraction module M. TFBS It consists of two feature extraction units (FEU) and two upsampling modules (N). US The structure of each feature extraction unit (FEU) consists of: a downsampling module M. DS and offset attention and residual module M OAR ,like Figure 3 As shown, the single-frame Transformer spatial feature extraction module M TFBS Spatial scale features are extracted according to the following steps. of:

[0059] 1) The downsampling module M in the first feature extraction unit (FEU) DS For the aggregated local neighborhood features F t Sampling at the farthest point, using distant points as keypoints, can increase the effective receptive field and establish direct long-distance dependencies, thus obtaining the features of the sampled points.

[0060] Use the KNN algorithm to analyze the features of the sampling points. The query is performed to obtain the grouped index, and then the grouped index is aggregated and projected before Max Pooling is performed to obtain the downsampling module M. DS Output downsampled features

[0061] 2) Downsampling features Input offset attention and residual module M OAR After feature extraction, we obtain the effective aggregated long-range contextual features F′. t And the features output by the first feature extraction unit FEU;

[0062] 3) Contextual features F′ t Input the first upsampling module M US In the middle, after projection through the LN linear layer, interpolation is performed to obtain the first-level features after segmentation.

[0063] 4) Contextual features F′ t After being processed by the second feature extraction unit (FEU), the context features F″ output by the second feature extraction unit are obtained. t ;

[0064] 5) Contextual features F″ t Input the second upsampling module M US After processing, the segmented second-level features are obtained.

[0065] 6) The first-level features after segmentation Second-level features Connecting along dimensions yields spatial scale features with multi-level characteristics.

[0066] Time Feature Extraction Module M TRT Using consecutive frames of 3D point cloud data (x) from frame t-τ to frame t-1 t-τ ,…,x t-2 ,x t-1 () is used as input and passed through the multi-frame point embedding module M PED Convolutional processing of core points in point clouds to aggregate consecutive multi-frame 3D point cloud data (x t-τ ,…,x t-2 ,x t-1 The local geometry and contextual information of the region are used to obtain the local neighborhood features (F) aggregated from frame t-τ to frame t-1. t-τ ,…,F t-2 ,F t-1 );

[0067] The local neighborhood features (F) of consecutive frames aggregated from frame t-τ to frame t-1 t-τ ,…,F t-2 ,F t-1 Input multi-frame Transformer spatial feature extraction module M TFBDS After feature extraction, the spatial scale features from frame t-τ to frame t-1 are obtained.

[0068] Spatial scale features from frame t-τ to frame t-1 Spatial scale features of frame t Common input multi-frame Transformer temporal feature extraction module M TFBDT The time-scale features of frame t are obtained through processing.

[0069] Step 2.2: Concatenate the spatial scale features of the t-th frame by channel. and the temporal scale features of the t-th frame Obtain the spatiotemporal features of frame t Spatiotemporal characteristics That is, the 3D point cloud data x of the t-th frame in the 3D point cloud video. t The salient features;

[0070] Step 2.3: A 3D point cloud video dynamics detection model based on fov viewpoint prediction, including: an LSTM module and a fov viewpoint embedding module M. PEFand Transformer viewpoint feature extraction module M TFBF ;

[0071] 3D point cloud video dynamics detection model with historical FOV viewpoint information As input, and after viewpoint prediction by the LSTM module, we obtain a×b FOV viewpoint prediction information for a users watching b frames of point cloud video. in, This represents the FOV viewpoint prediction information when user a watches the point cloud video of frame b.

[0072] Prediction information from a×b FOV viewpoints The FOV viewpoint information of the m-th user watching the t-th frame of the point cloud video is randomly selected. Then extract the FOV viewpoint information. Point cloud data x′ m,t , where m∈(1,a);

[0073] Point cloud data x′ m,t Input fov viewpoint embedding module M PEF Perform convolution on the core points of the point cloud to aggregate the point cloud data x′. m,t Local geometry and contextual information are used to obtain x′ m,t Aggregated local neighborhood features F m,t ;

[0074] The aggregated local neighborhood features F m,t Input Transformer viewpoint feature extraction module M TFBF Feature extraction is performed to obtain the predicted point cloud features of the viewpoint when the m-th user watches the point cloud video of frame t. And serve as a dynamic feature when the m-th user views the t-th frame of the point cloud video;

[0075] Step 2.4, saliency features of the point cloud video in frame t. Dynamic features of the point cloud video of the m-th user at frame t The common input salient features and dynamic features are fused in the fusion model:

[0076] 1) Utilize Fourier transform to analyze the saliency features of the point cloud video in frame t. Dynamic features of the point cloud video in frame t After averaging, the initial total feature F″′, which reflects the combined level of dynamic and significance characteristics, is obtained. m,t ;

[0077] 2) The initial total feature F″′ m,t Perform region aggregation processing to obtain the region aggregation feature F″.m,t ;

[0078] 3) Aggregate the regional features F″ m,t Normalization is performed to obtain the normalized region aggregation features. Then, serialization is performed to obtain the total feature F′ of frame t. m,t

[0079] Step 2.5: Decode the network using the total features F′ of the m-th user in the t-th frame. m,t As input, after passing through the decoding network, the final predicted FOV viewpoint image of the m-th user in the t-th frame is obtained;

[0080] Step 3: Model Training

[0081] Based on b-frame point cloud data (x1, x2, ..., x t ,…,x b The gradient descent method is used to train the fov viewpoint prediction model for 3D point cloud video streaming transmission based on Transformer. The Diou loss is used as the loss function to adjust the parameters of the model. The training stops when the number of training iterations reaches the set number or the loss function tends to a stable value, thus obtaining the optimal fov viewpoint prediction model for predicting the user's fov viewpoint.

[0082] The specific process of FOV viewpoint prediction in step three is as follows: after preprocessing the historical point cloud video data, it is fed into the trained FOV viewpoint prediction network model to predict the FOV viewpoint of the user watching the point cloud video frames. The above prediction process is repeated until the prediction of all point cloud video frames is completed.

[0083] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0084] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

[0085] In summary, this invention proposes for the first time a point cloud video saliency detection method based on Transformer. It utilizes a point embedding module to aggregate the local neighborhood features and long-range contextual information of each point in the point cloud video, while using a Transformer feature extraction module to extract the spatiotemporal features of the point cloud video. Furthermore, it combines Transformer-based saliency detection with dynamic detection based on the user's FOV viewpoint to obtain the final prediction of the user's FOV viewpoint for a given frame, thus achieving a high prediction accuracy.

Claims

1. A method for predicting the viewpoint of 3D point cloud video streaming transmission based on Transformer, characterized in that, The procedure is as follows: Step 1: Collection and preprocessing of viewpoint data: Step 1.1: Collect and extract a×b historical FOV viewpoint information from a users watching b consecutive frames of point cloud video. in, This represents the FOV viewpoint information when the a-th user views the b-th frame of the point cloud video; Step 1.2: Extract the a×b historical FOV viewpoint information. b-frame point cloud data (x1, x2, ..., x) from the user's perspective t , ..., x b ), where x t Let t represent the fovo viewpoint information of a users watching the t-th frame of the point cloud video, where t∈(1,b); Step 2: Construct a viewpoint prediction model for 3D point cloud video streaming transmission based on Transformer, including: a 3D point cloud video saliency detection model based on Transformer, a 3D point cloud video dynamics detection model based on foveo viewpoint prediction, a saliency feature and dynamic feature fusion module, and a decoding network. Step 2.1: The Transformer-based 3D point cloud video saliency detection model includes: a Transformer-based spatial feature extraction module M. TRS and Transformer-based temporal feature extraction module M TRT ; Among them, the spatial feature extraction module M TRS Includes: Single-frame point embedding module M PE and the single-frame Transformer spatial feature extraction module M TFBS The time feature extraction module M TRT Includes: Multi-frame point embedding module M PED Multi-frame Transformer spatial feature extraction module M TFBDS and the multi-frame Transformer temporal feature extraction module M TFBDT ; The spatial feature extraction module M TRS Using the 3D point cloud data x of frame t t As input, and passed through the single-frame point embedding module M PE The point cloud core points are processed by convolution to aggregate the 3D point cloud data x of the t-th frame. t The local geometry and contextual information are used to obtain the aggregated local neighborhood features F of frame t. t ; The aggregated local neighborhood features F t Input the single-frame Transformer spatial feature extraction module M TFBS After feature extraction, the spatial scale features of frame t are obtained. The time feature extraction module M TRT Using consecutive frames of 3D point cloud data (x) from frame t-τ to frame t-1 t-τ , ..., x t-2 x t-1 The input is used as the multi-frame point embedding module M. PED Convolutional processing of core points in point clouds to aggregate consecutive multi-frame 3D point cloud data (x t-τ , ..., x t-2 x t-1 The local geometry and contextual information of the region are used to obtain the local neighborhood features (F) aggregated from frame t-τ to frame t-1. t-τ F t-2 F t-1 ); The local neighborhood features (F) of consecutive frames aggregated from frame t-τ to frame t-1 t-τ F t-2 F t-1 Input the multi-frame Transformer spatial feature extraction module M TFBDS After feature extraction, the spatial scale features from frame t-τ to frame t-1 are obtained. Spatial scale features from frame t-τ to frame t-1 Spatial scale features of frame t The data is input into the multi-frame Transformer temporal feature extraction module MTFBDT for processing to obtain the temporal scale features of the t-th frame. Step 2.2: Concatenate the spatial scale features of the t-th frame by channel. and the temporal scale features of the t-th frame Obtain the spatiotemporal features of frame t The spatiotemporal features That is, the 3D point cloud data x of the t-th frame in the 3D point cloud video. t The salient features; Step 2.3: The 3D point cloud video dynamics detection model based on fov viewpoint prediction includes: an LSTM module and a fov viewpoint embedding module M. PEF and Transformer viewpoint feature extraction module M TFBF ; The 3D point cloud video dynamic detection model uses the historical FOV viewpoint information. As input, and after viewpoint prediction by the LSTM module, a×b FOV viewpoint prediction information is obtained when a users watch b frames of point cloud video. in, This represents the FOV viewpoint prediction information when user a watches the point cloud video of frame b. Prediction information from the a×b FOV viewpoints The FOV viewpoint information of the m-th user watching the t-th frame of the point cloud video is randomly selected. Then, the fov viewpoint information is extracted. Point cloud data x′ m，t , where m∈(1,a); The point cloud data x′ m，t Input the fov viewpoint embedding module M PEF Perform convolution on the core points of the point cloud to aggregate the point cloud data x′. m，t Local geometry and contextual information are used to obtain x′ m，t Aggregated local neighborhood features F m，t ; The aggregated local neighborhood features F m，t Input the Transformer viewpoint feature extraction module M TFBF Feature extraction is performed to obtain the predicted point cloud features of the viewpoint when the m-th user watches the point cloud video of frame t. And serve as a dynamic feature when the m-th user views the t-th frame of the point cloud video; Step 2.4, the saliency features of the point cloud video in frame t. Dynamic features of the point cloud video of the m-th user at frame t After fusion processing in the input saliency feature and dynamic feature fusion model, the total feature F′ of the m-th user in the t-th frame is obtained. m，t ; Step 2.5: The decoding network uses the total features F′ of the m-th user in the t-th frame. m，t As input, after passing through the decoding network, the final predicted FOV viewpoint image of the m-th user in the t-th frame is obtained; Step 3: Model Training Based on b-frame point cloud data (x1, x2, ..., x...) t , ..., x b The gradient descent method is used to train the fov viewpoint prediction model for 3D point cloud video streaming transmission based on Transformer, and the Diou loss is used as the loss function to adjust the parameters of the model. When the number of training iterations reaches the set number or the loss function tends to a stable value, the training stops, thereby obtaining the optimal fov viewpoint prediction model, which is used to predict the user's fov viewpoint.

2. The foveation viewpoint prediction method for 3D point cloud video streaming transmission based on Transformer as described in claim 1, characterized in that, The single-frame Transformer spatial feature extraction module M in step two TFBS It consists of two feature extraction units (FEU) and two upsampling modules (M). US The structure, each feature extraction unit (FEU) consists of: downsampling module M DS The offset attention and residual module MOAR are used, and spatial scale features are extracted according to the following steps. 1) The downsampling module M in the first feature extraction unit (FEU) DS The aggregated local neighborhood features F t Perform farthest point sampling to obtain sampling point features. Use the KNN algorithm to analyze the features of the sampling points. A query is performed to obtain the grouping index, and the grouping index is aggregated and projected before Max Pooling is applied to obtain the downsampling module M. DS Output downsampled features 2) The downsampling features Input the offset attention and residual module M OAR After feature extraction, we obtain the effective aggregated long-range contextual features F′. t And the features output by the first feature extraction unit FEU; 3) The context feature F′ t Input the first upsampling module M US In the middle, after projection through the LN linear layer, interpolation is performed to obtain the first-level features after segmentation. 4) The context feature F′ t After being processed by the second feature extraction unit (FEU), the context features F″ output by the second feature extraction unit are obtained. t ; 5) The context feature F″ t Input the second upsampling module M US After processing, the segmented second-level features are obtained. 6) The first-level features after segmentation Second-level features Connecting along dimensions yields spatial scale features with multi-level characteristics.

3. The FOV (viewpoint prediction) method for 3D point cloud video streaming transmission based on Transformer as described in claim 1, characterized in that, The fusion model of salient features and dynamic features in step 2.4 is performed according to the following steps: 1) Utilize Fourier transform to analyze the saliency features of the point cloud video in frame t. Dynamic features of the point cloud video in frame t After averaging, the initial total feature F″′, which reflects the combined level of dynamic and significance characteristics, is obtained. m，t ; 2) The initial total feature F″′ m，t Perform region aggregation processing to obtain the region aggregation feature F″. m，t ; 3) Aggregate the region features F″ m，t Normalization is performed to obtain the normalized region aggregation feature F′. m，t Then, serialization is performed to obtain the total feature F′ of frame t. m，t .

4. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store programs that support the processor in executing any of the FOV viewpoint prediction methods of claims 1-3, and the processor is configured to execute the programs stored in the memory.

5. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program, when run by a processor, performs the steps of any of the FOV viewpoint prediction methods described in claims 1-3.

Images