Viewport trajectory prediction model training method, viewport trajectory prediction method and device

Abstract

The application provides a viewport trajectory prediction model training method, a viewport trajectory prediction method and a device. The method comprises: obtaining labeled point cloud video data and unlabeled point cloud video data; training an initial viewport trajectory prediction model through the labeled point cloud video data to obtain a first viewport trajectory prediction model; training the first viewport trajectory prediction model through the unlabeled point cloud video data and a preset space-time constraint condition to obtain a second viewport trajectory prediction model; if the second viewport trajectory prediction model does not satisfy a preset convergence condition, taking the second viewport trajectory prediction model as the initial viewport trajectory prediction model, and jumping to the step of training the initial viewport trajectory prediction model through the labeled point cloud video data for execution until the second viewport trajectory prediction model satisfies the preset convergence condition, and taking the second viewport trajectory prediction model as a trained viewport trajectory prediction model. The application can improve the adaptability of the viewport trajectory prediction model.

Description

Technical Field

[0001] This application relates to the field of trajectory prediction technology, and in particular to a viewport trajectory prediction model training method, viewport trajectory prediction method and device. Background Technology

[0002] Point cloud video (PCV), a multimedia technology based on 3D data, is widely used in virtual reality, augmented reality, and metaverse scenarios, supporting free interaction by users in a six-degrees-of-freedom (6-DoF) environment. Viewport trajectory prediction technology, a key component of point cloud video, predicts the user's viewport trajectory. This trajectory refers to the set of positions and directions formed by the user's line of sight in 3D space over time as they watch point cloud video, reflecting the path of the user's changing perspective. This allows the system to preload and render the predicted viewport content, reducing the time users spend waiting for data loading and processing when switching perspectives, resulting in a smoother and more natural viewing experience.

[0003] In related technologies, the training of viewport trajectory prediction models mainly relies on supervised learning. However, supervised learning primarily uses a large amount of high-quality labeled data. Since labeled data is usually concentrated in specific environments or user behavior patterns, the prediction performance of viewport trajectory prediction models drops significantly when faced with complex and ever-changing application scenarios or new user behaviors, making it difficult to accurately adapt to new situations. Therefore, existing viewport trajectory prediction models suffer from weak adaptability. Summary of the Invention

[0004] The main objective of this application is to propose a viewport trajectory prediction model training method, viewport trajectory prediction method and device, which aims to solve the problem that existing viewport trajectory prediction models have weak adaptability.

[0005] To achieve the above objectives, a first aspect of this application proposes a viewport trajectory prediction model training method, the method comprising:

[0006] Obtain labeled point cloud video data and unlabeled point cloud video data;

[0007] The initial viewport trajectory prediction model is trained using the labeled point cloud video data to obtain the first viewport trajectory prediction model.

[0008] The first viewport trajectory prediction model is trained using the unlabeled point cloud video data and preset spatiotemporal constraints to obtain the second viewport trajectory prediction model. The preset spatiotemporal constraints include at least one of a gaze-following loss condition, a smooth motion loss condition, and a context-dependent loss condition. The gaze-following loss condition is used to adjust the difference between the predicted points of the first viewport trajectory prediction model and the real points. The smooth motion loss condition is used to indicate the spatial coherence of the predicted points of the first viewport trajectory prediction model. The context-dependent loss condition is used to indicate the temporal consistency of the predicted points of the first viewport trajectory prediction model.

[0009] If the second viewport trajectory prediction model does not meet the preset convergence condition, then the second viewport trajectory prediction model is used as the initial viewport trajectory prediction model, and the step of training the initial viewport trajectory prediction model with the labeled point cloud video data to obtain the first viewport trajectory prediction model is re-executed; and the step of training the first viewport trajectory prediction model with the unlabeled point cloud video data and preset spatiotemporal constraints to obtain the second viewport trajectory prediction model is re-executed until the obtained second viewport trajectory prediction model meets the preset convergence condition, and the second viewport trajectory prediction model is used as the trained viewport trajectory prediction model.

[0010] In some implementations, the preset constraints include the gaze-following loss condition;

[0011] The step of training the first viewport trajectory prediction model using the unlabeled point cloud video data and preset spatiotemporal constraints to obtain the second viewport trajectory prediction model includes:

[0012] Key features are extracted from each video frame in the unlabeled point cloud video data to obtain multiple key features corresponding to each video frame;

[0013] For each video frame, perform:

[0014] Each of the key features is calculated to obtain an attention score corresponding to each key feature; based on the attention scores corresponding to the key features, the predicted attention distribution of the video frame is obtained.

[0015] The gaze following loss value is calculated based on the predicted attention distribution corresponding to each video frame and the gaze point following loss condition;

[0016] The first viewport trajectory prediction model is trained based on the gaze-following loss value to obtain the second viewport trajectory prediction model.

[0017] In some implementations, the preset constraint conditions include the smooth motion loss conditions;

[0018] The step of training the first viewport trajectory prediction model using the unlabeled point cloud video data and preset spatiotemporal constraints to obtain the second viewport trajectory prediction model includes:

[0019] Obtain the predicted position of each video frame in the unlabeled point cloud video data by the first viewport trajectory prediction model;

[0020] For each video frame, execute:

[0021] The speed change value of the video frame is calculated based on the predicted position of the video frame, the predicted position of the preceding video frame, and the predicted position of the second preceding video frame.

[0022] The smooth motion loss value is calculated based on the speed change value of each video frame and the smooth motion loss condition.

[0023] The first viewport trajectory prediction model is trained based on the smooth motion loss value to obtain the second viewport trajectory prediction model.

[0024] In some implementations, the preset constraints include the context-dependent loss condition;

[0025] The step of training the first viewport trajectory prediction model using the unlabeled point cloud video data and preset spatiotemporal constraints to obtain the second viewport trajectory prediction model includes:

[0026] Obtain the predicted position of each video frame in the unlabeled point cloud video data by the first viewport trajectory prediction model;

[0027] Context features are extracted from each video frame in the unlabeled point cloud video data to obtain the context features of each video frame;

[0028] For each video frame, execute:

[0029] The context-dependent loss value of the video frame is calculated based on the predicted position of the next video frame and the context features of the video frame.

[0030] The context dependency loss value of the unlabeled point cloud video data is calculated based on the context dependency loss value of each video frame and the context dependency loss condition.

[0031] The first viewport trajectory prediction model is trained based on the context dependency loss value of the unlabeled point cloud video data to obtain the second viewport trajectory prediction model.

[0032] To achieve the above objectives, a second aspect of this application proposes a viewport trajectory prediction method, the method comprising:

[0033] Acquire the point cloud video data to be inspected;

[0034] The point cloud video data to be detected is input into the viewport trajectory prediction model, and the viewport trajectory prediction model outputs a set of predicted viewport trajectories. The set of predicted viewport trajectories includes multiple viewport trajectories predicted by the viewport trajectory prediction model. The viewport trajectory prediction model is obtained using the viewport trajectory prediction model training method described in the first aspect.

[0035] The target viewport trajectory set is obtained by filtering multiple viewport trajectories in the predicted viewport trajectory set.

[0036] In some implementations, the set of predicted viewport trajectories may also include the probability corresponding to each viewport trajectory;

[0037] The step of filtering multiple viewport trajectories in the predicted viewport trajectory set to obtain the target viewport trajectory set includes:

[0038] The entropy of the predicted viewport trajectory set is calculated based on the probability corresponding to each viewport trajectory in the predicted viewport trajectory set.

[0039] The number of viewport trajectories contained in the target trajectory set is determined based on the entropy of the predicted viewport trajectory set.

[0040] The target viewport trajectory set is formed by selecting the number of viewport trajectories from the predicted viewport trajectory set.

[0041] To achieve the above objectives, a third aspect of this application provides a viewport trajectory prediction model training apparatus, the apparatus comprising:

[0042] The data acquisition module is used to acquire labeled point cloud video data and unlabeled point cloud video data;

[0043] The first training module is used to train the initial viewport trajectory prediction model using the labeled point cloud video data to obtain the first viewport trajectory prediction model.

[0044] The second training module is used to train the first viewport trajectory prediction model using the unlabeled point cloud video data and preset spatiotemporal constraints to obtain a second viewport trajectory prediction model. The preset spatiotemporal constraints include at least one of a gaze-following loss condition, a smooth motion loss condition, and a context-dependent loss condition. The gaze-following loss condition is used to adjust the difference between the predicted points of the first viewport trajectory prediction model and the real points. The smooth motion loss condition is used to indicate the spatial coherence of the predicted points of the first viewport trajectory prediction model. The context-dependent loss condition is used to indicate the temporal consistency of the predicted points of the first viewport trajectory prediction model.

[0045] The third training module is used to, if the second viewport trajectory prediction model does not meet the preset convergence condition, take the second viewport trajectory prediction model as the initial viewport trajectory prediction model and re-execute the step of training the initial viewport trajectory prediction model with the labeled point cloud video data to obtain the first viewport trajectory prediction model; and re-execute the step of training the first viewport trajectory prediction model with the unlabeled point cloud video data and preset spatiotemporal constraints to obtain the second viewport trajectory prediction model, until the obtained second viewport trajectory prediction model meets the preset convergence condition, and take the second viewport trajectory prediction model as the trained viewport trajectory prediction model.

[0046] To achieve the above objectives, a fourth aspect of this application provides a viewport trajectory prediction device, the device comprising:

[0047] The point cloud acquisition module is used to acquire the point cloud video data to be detected.

[0048] The trajectory prediction module is used to input the point cloud video data to be detected into the viewport trajectory prediction model, and the viewport trajectory prediction model outputs a set of predicted viewport trajectories. The set of predicted viewport trajectories includes multiple viewport trajectories predicted by the viewport trajectory prediction model. The viewport trajectory prediction model is obtained by the viewport trajectory prediction model training method described in the first aspect.

[0049] The trajectory filtering module is used to filter multiple viewport trajectories in the predicted viewport trajectory set to obtain the target viewport trajectory set.

[0050] To achieve the above objectives, a fifth aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the viewport trajectory prediction model training method as described in the first aspect, and / or the viewport trajectory prediction method as described in the second aspect.

[0051] To achieve the above objectives, a sixth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the viewport trajectory prediction model training method as described in the first aspect, and / or the viewport trajectory prediction method as described in the second aspect.

[0052] To achieve the above objectives, a seventh aspect of this application provides a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to implement the viewport trajectory prediction model training method as described in the first aspect, and / or the viewport trajectory prediction method as described in the second aspect.

[0053] The viewport trajectory prediction model training method, viewport trajectory prediction method, and apparatus proposed in this application train an initial viewport trajectory prediction model using labeled point cloud video data. This allows the initial viewport trajectory prediction model to learn basic viewport trajectory prediction rules, resulting in a first viewport trajectory prediction model. The first viewport trajectory prediction model is then trained using unlabeled point cloud video data, with preset spatiotemporal constraints added during training to make the prediction results of the first viewport trajectory prediction model more consistent with natural spatiotemporal rules, resulting in a second viewport trajectory prediction model. If the second viewport trajectory prediction model does not meet preset convergence conditions, the above training process is repeated. By alternating between labeled and unlabeled data in each training iteration, the trained viewport trajectory prediction model can not only accurately predict the viewport trajectory of labeled point cloud video data but also extract useful information from unlabeled data to predict the viewport trajectory of unlabeled point cloud video data. This results in a better generalization ability and robustness of the trained viewport trajectory prediction model, thus improving its adaptability. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating a viewport trajectory prediction model training method provided in an embodiment of this application;

[0055] Figure 2 This is a flowchart illustrating the viewport trajectory prediction method provided in an embodiment of this application;

[0056] Figure 3 This is a schematic diagram of the structure of the viewport trajectory prediction model training device provided in the embodiments of this application;

[0057] Figure 4 This is a schematic diagram of the viewport trajectory prediction device provided in the embodiments of this application;

[0058] Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0062] Point cloud video (PCV), a multimedia technology based on 3D data, is widely used in virtual reality, augmented reality, and metaverse scenarios, supporting free interaction by users in a six-degrees-of-freedom (6-DoF) environment. Viewport trajectory prediction technology, a key component of point cloud video, predicts the user's viewport trajectory. This trajectory refers to the set of positions and directions formed by the user's line of sight in 3D space over time as they watch point cloud video, reflecting the path of the user's changing perspective. This allows the system to preload and render the predicted viewport content, reducing the time users spend waiting for data loading and processing when switching perspectives, resulting in a smoother and more natural viewing experience.

[0063] In related technologies, the training of viewport trajectory prediction models mainly relies on supervised learning. However, supervised learning primarily uses a large amount of high-quality labeled data. Since labeled data is usually concentrated in specific environments or user behavior patterns, the prediction performance of viewport trajectory prediction models drops significantly when faced with complex and ever-changing application scenarios or new user behaviors, making it difficult to accurately adapt to new situations. Therefore, existing viewport trajectory prediction models suffer from weak adaptability.

[0064] Based on this, the embodiments of this application provide a viewport trajectory prediction model training method, viewport trajectory prediction method and apparatus, which aim to solve the problem that the existing viewport trajectory prediction models have weak adaptability.

[0065] The viewport trajectory prediction model training method, viewport trajectory prediction method and apparatus provided in this application are specifically described through the following embodiments. First, the viewport trajectory prediction model training method in this application embodiment is described.

[0066] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0067] Foundational artificial intelligence technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0068] The viewport trajectory prediction model training method provided in this application relates to the technical field of trajectory prediction. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the viewport trajectory prediction model training method, but is not limited to the above forms.

[0069] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0070] It should be noted that in all specific embodiments of this application, when processing data related to identity or characteristics, such as information, behavioral data, historical data, and location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when this application requires the acquisition of sensitive personal information, separate permission or consent is obtained through pop-ups or redirects to confirmation pages. Only after obtaining explicit separate permission or consent is the necessary data required for the proper functioning of this application's embodiments obtained.

[0071] Figure 1 This is a flowchart illustrating the viewport trajectory prediction model training method provided in this application embodiment. Please refer to [link / reference]. Figure 1 The viewport trajectory prediction model training method provided in this application embodiment may include, but is not limited to, steps S101 to S104.

[0072] Step S101: Obtain labeled point cloud video data and unlabeled point cloud video data.

[0073] In this step, both labeled and unlabeled point cloud video data can be obtained from existing datasets. For example, labeled point cloud video data can be obtained from the S3DIS dataset; unlabeled point cloud video data can be obtained from the ScanNet dataset and / or the Matterport3D dataset.

[0074] Step S102: Train the initial viewport trajectory prediction model using the labeled point cloud video data to obtain the first viewport trajectory prediction model.

[0075] In this step, the initial viewport trajectory prediction model includes a backbone network and a head network. Specifically, the backbone network is a 3D convolutional network consisting of five convolutional blocks, each containing a 3D convolutional layer with progressively increasing filter sizes (32, 64, 128, 256, 512). Batch normalization and ReLU activation functions are applied, and each convolutional block contains residual connections. The backbone network extracts features from the point cloud video data. The head network includes a global pooling layer, a bidirectional long short-term memory (BiLSTM) layer, and fully connected layers. The global pooling layer aggregates spatial features, and then the BiLSTM layer models temporal dynamics based on the aggregated spatial features. The output of the BiLSTM layer is then passed through a series of fully connected layers with dropout (a regularization technique for neural networks used to randomly drop out a portion of neurons during training to prevent overfitting).

[0076] Furthermore, the initial viewport trajectory prediction model is trained in a supervised manner using labeled point cloud video data. During supervised training, the mean squared error (MSE) function can be used as the loss function to obtain a first viewport trajectory prediction model that has learned the basic viewport trajectory prediction rules. The model training process for the initial viewport trajectory prediction model using labeled point cloud video data is similar to existing supervised training processes and will not be elaborated upon here.

[0077] Step S103: Train the first viewport trajectory prediction model using the unlabeled point cloud video data and preset spatiotemporal constraints to obtain the second viewport trajectory prediction model.

[0078] In this step, the first viewport trajectory prediction model is trained unsupervised using unlabeled point cloud video data. During the training process, the loss value is calculated using preset spatiotemporal constraints. The parameters of the first viewport trajectory model are adjusted through backpropagation to obtain the second viewport trajectory prediction model.

[0079] The preset spatiotemporal constraints include at least one of the following: gaze point following loss condition, smooth motion loss condition, and context-dependent loss condition. The gaze point following loss condition is used to adjust the difference between the predicted point and the real point of the first viewport trajectory prediction model. The smooth motion loss condition is used to indicate the spatial coherence of the predicted point of the first viewport trajectory prediction model. The context-dependent loss condition is used to indicate the temporal consistency of the predicted point of the first viewport trajectory prediction model.

[0080] Step S104: If the second viewport trajectory prediction model does not meet the preset convergence condition, then the second viewport trajectory prediction model is used as the initial viewport trajectory prediction model, and the step of training the initial viewport trajectory prediction model with the labeled point cloud video data to obtain the first viewport trajectory prediction model is re-executed; and the step of training the first viewport trajectory prediction model with the unlabeled point cloud video data and the preset spatiotemporal constraints to obtain the second viewport trajectory prediction model is re-executed until the obtained second viewport trajectory prediction model meets the preset convergence condition, and the second viewport trajectory prediction model is used as the trained viewport trajectory prediction model.

[0081] In this step, if the second viewport trajectory prediction model does not meet the preset convergence condition, the second viewport trajectory model is used as the initial viewport trajectory model. The steps of training the initial viewport trajectory prediction model with labeled point cloud video data and training the first viewport trajectory prediction model with unlabeled point cloud video data and preset spatiotemporal constraints are repeated until the second viewport trajectory prediction model meets the prediction convergence condition.

[0082] The preset convergence condition can be set to a preset number of iterations, i.e., when the model iterations reach the preset number of iterations, the iteration stops; it can be set to the total training loss threshold, which can be obtained by weighted summation of supervised training loss and unsupervised training loss, i.e., when the total training loss is less than the total training loss threshold, the iteration stops; or it can be set to satisfy either the preset number of iterations or the total training loss threshold, without any limitation here.

[0083] In this implementation, an initial viewport trajectory prediction model is trained using labeled point cloud video data. This allows the initial viewport trajectory prediction model to learn basic viewport trajectory prediction rules, resulting in a first viewport trajectory prediction model. Then, the first viewport trajectory prediction model is trained using unlabeled point cloud video data, with preset spatiotemporal constraints added during training to make the prediction results of the first viewport trajectory prediction model more consistent with natural spatiotemporal patterns, resulting in a second viewport trajectory prediction model. If the second viewport trajectory prediction model does not meet the preset convergence conditions, the above training process is repeated. By alternating between labeled and unlabeled data in each training iteration, the trained viewport trajectory prediction model can not only accurately predict the viewport trajectory of labeled point cloud video data but also extract useful information from unlabeled data to predict the viewport trajectory of unlabeled point cloud video data. This results in a better generalization ability and robustness of the trained viewport trajectory prediction model, thus improving its adaptability.

[0084] In some implementations, the preset constraints include the gaze point following loss condition. The first viewport trajectory prediction model is trained in step S103 using the unlabeled point cloud video data and the preset spatiotemporal constraints to obtain the second viewport trajectory prediction model, which may include, but is not limited to, steps S201 to S204.

[0085] Step S201: Extract key features from each video frame in the unlabeled point cloud video data to obtain multiple key features corresponding to each video frame.

[0086] Step S202: For each video frame, perform the following:

[0087] Each of the key features is calculated to obtain an attention score corresponding to each key feature; based on the attention scores corresponding to the key features, the predicted attention distribution of the video frame is obtained.

[0088] Step S203: Calculate the gaze following loss value based on the predicted attention distribution corresponding to each video frame and the gaze point following loss condition.

[0089] Step S204: Train the first viewport trajectory prediction model based on the gaze-following loss value to obtain the second viewport trajectory prediction model.

[0090] In this implementation, since not all regions have the same salience for predicting viewport trajectories in practical applications, users usually focus on certain important regions in the scene, which may be driven by visual salience or contextual information. In order to capture this behavior, this application uses a gaze-following loss condition to make the viewport trajectory predicted by the first viewport trajectory prediction model aligned with the region that the user is most likely to focus on.

[0091] Specifically, the backbone network in the first viewport trajectory model extracts key features from each video frame of the unlabeled point cloud video data, obtaining multiple key features corresponding to each video frame. These key features include spatial and temporal features, such as object boundaries, motion, and contrast. Dot product attention or additive attention methods can be used to calculate the attention score for each key feature. The attention score corresponding to each key feature is then transformed into a probability distribution using the Softmax function, ensuring that the sum of the attention values ​​for all regions is 1. Based on the probability distribution corresponding to each key feature, the predicted attention distribution for each video frame is obtained. The gaze-following loss value is calculated based on the predicted attention distribution of each video frame in the unlabeled point cloud video data and the gaze-following loss condition. Then, through backpropagation, the parameters of the first viewport trajectory prediction model are adjusted using the gaze-following loss value to obtain the second viewport trajectory prediction model.

[0092] The gaze-following loss condition can be expressed by Equation 1:

[0093] (1);

[0094] In the formula, This represents the fixation-following loss value. N This indicates the number of video frames in the unlabeled point cloud video data. i Indicates the first i Video frames, Indicates the first i Predictive attention distribution of video frames, Indicates the first i The true saliency distribution of video frames.

[0095] It should be noted that the true saliency distribution of each video frame can be obtained using the user's eye-tracking data or a saliency detection algorithm.

[0096] In this implementation, the first viewport trajectory model is trained using unlabeled point cloud video data to enhance the model's generalization ability and robustness. Furthermore, a gaze-following loss condition is incorporated during training, projecting the viewport prediction onto a low-dimensional subspace defined by the key attention region. This allows the viewport trajectory prediction model to learn to prioritize processing key regions, ensuring that the predicted viewport trajectory aligns with the user's natural viewing pattern. Consequently, the viewport trajectory prediction model achieves more accurate and robust viewport trajectory prediction in various environments, thereby improving the accuracy of the viewport trajectory prediction model's predictions.

[0097] In some implementations, the preset constraints include smooth motion loss conditions. The training of the first viewport trajectory prediction model in step S103 using the unlabeled point cloud video data and the preset spatiotemporal constraints to obtain the second viewport trajectory prediction model may include, but is not limited to, steps S301 to S304.

[0098] Step S301: Obtain the predicted position of each video frame in the unlabeled point cloud video data by the first viewport trajectory prediction model.

[0099] Step S302: For each video frame, execute:

[0100] The speed change value of the video frame is calculated based on the predicted position of the video frame, the predicted position of the preceding video frame, and the predicted position of the second preceding video frame.

[0101] Step S303: Calculate the smooth motion loss value based on the speed change value of each video frame and the smooth motion loss condition.

[0102] Step S304: Train the first viewport trajectory prediction model based on the smooth motion loss value to obtain the second viewport trajectory prediction model.

[0103] In this implementation, predicting a realistic viewport trajectory in immersive environments such as virtual reality requires predicting where the user might look at at any given moment. Natural human movement is smooth and continuous, and is affected by physical constraints (such as the inertia of the head and eyes). Sudden and irregular changes in the viewport position are not only rare, but may also cause user discomfort and greatly affect the user experience. Therefore, this application sets a smooth movement loss to ensure the continuity and smoothness of the motion by penalizing drastic changes in velocity and acceleration between adjacent frames.

[0104] Furthermore, the motion modeling of the viewport is a position time series across video frames, where the predicted position of each frame depends on the position information of its neighboring frames. Since sudden changes in the prediction can lead to abrupt changes in velocity, in order to minimize such abrupt changes in velocity, this application measures the smoothness or abruptness of the transition between positions by calculating the velocity difference between three consecutive video frames.

[0105] Specifically, the predicted position of each video frame in the unlabeled point cloud video data is obtained by the first viewport trajectory prediction model. For each video frame, the velocity change value of the video frame is obtained based on the preceding video frame, the second preceding video frame, and the predicted position of the video frame. The smoothing motion loss value is calculated by using the velocity change value of each video frame in the unlabeled point cloud video data and the smoothing motion loss condition. Then, the parameters of the first viewport trajectory prediction model are adjusted by backpropagation using the smoothing motion loss value to obtain the second viewport trajectory prediction model.

[0106] The speed change value for each video frame can be calculated using the following formula 2:

[0107] (2);

[0108] In the formula, The speed change value of video frames, i Indicates the first i Video frames, t Indicates the time step. Indicates the first i Predicted position of video frame Indicates the first iThe predicted position of the preceding video frame. Indicates the first i The predicted position of the first two video frames.

[0109] The loss condition for smooth motion can be expressed by the following equation 3:

[0110] (3);

[0111] In the formula, This represents the smooth motion loss value. N This indicates the number of video frames in the unlabeled point cloud video data. t Indicates the time step. i Represents the i-th video frame. This represents the speed change value of the i-th video frame.

[0112] In this implementation, unsupervised smoothness constraint training of the viewport trajectory prediction model is performed using unlabeled point cloud video data. This helps the viewport trajectory prediction model learn more universally applicable motion laws, rather than relying solely on noise or bias that may exist in the labeled data. Furthermore, by using smooth motion loss conditions, the viewport trajectory prediction model not only learns to accurately predict the viewport position, but also generates smooth and uninterrupted trajectories, thereby improving the overall user experience.

[0113] In some implementations, the preset constraints include context-dependent loss conditions. The training of the first viewport trajectory prediction model in step S103 using the unlabeled point cloud video data and the preset spatiotemporal constraints to obtain the second viewport trajectory prediction model may include, but is not limited to, steps S401 to S404.

[0114] Step S401: Obtain the predicted position of each video frame in the unlabeled point cloud video data by the first viewport trajectory prediction model.

[0115] Step S402: Extract context features from each video frame in the unlabeled point cloud video data to obtain the context features of each video frame.

[0116] Step S403: For each video frame, execute:

[0117] The context-dependent loss value of the video frame is calculated based on the predicted position of the next video frame and the context features of the video frame.

[0118] Step S404: Calculate the context dependency loss value of the unlabeled point cloud video data based on the context dependency loss value of each video frame and the context dependency loss condition.

[0119] Step S405: Train the first viewport trajectory prediction model based on the context dependency loss value of the unlabeled point cloud video data to obtain the second viewport trajectory prediction model.

[0120] In this implementation, since users' attention is not focused on isolated frames in immersive environments such as virtual reality, but rather smoothly shifts based on previous events and visual cues, this application sets a context-dependent loss condition to ensure that the viewport trajectory predicted by the viewport trajectory prediction model is consistent in time, so that users' attention flows in a logical and context-driven manner throughout the time series.

[0121] Specifically, the predicted position of each video frame in the unlabeled point cloud video data is obtained by the first viewport trajectory prediction model, and the contextual features of each video frame are extracted through the bidirectional long short-term memory layer in the first viewport trajectory prediction model.

[0122] For each video frame, the context dependency loss value of the video frame is calculated based on the predicted position of the next video frame and the context features of the video frame. The context dependency loss value of the unlabeled point cloud video data is obtained through the context dependency loss of each video frame. Then, the parameters of the first viewport trajectory prediction model are adjusted through backpropagation using the context dependency loss value of the unlabeled point cloud video data to obtain the second viewport trajectory prediction model.

[0123] The context-dependent loss condition can be expressed by Equation 4:

[0124] (4);

[0125] In the formula, This represents the context-dependent loss value. N This indicates the number of video frames in the unlabeled point cloud video data. T This represents the total number of time steps. t Indicates the time step. i Represents the i-th video frame. This indicates the predicted position of the next video frame after the i-th video frame. Indicates the first i Contextual features of video frames.

[0126] In this implementation, by combining the predicted position output by the viewport trajectory prediction model with the context features extracted from the video frames and applying a context-dependent loss condition, the viewport trajectory prediction model is forced to learn to generate viewport trajectories that are more temporally coherent and more consistent with the context logic. This optimizes the first viewport trajectory prediction model and yields a second viewport trajectory prediction model with a smoother and more coherent predicted trajectory. This ensures that the trajectory predicted by the second viewport trajectory prediction model is not only accurate within each independent frame, but also consistent with the temporal structure of the entire interaction.

[0127] Based on the viewport trajectory prediction model obtained through training, this application provides a viewport trajectory prediction method. Figure 2 A flowchart illustrating a viewport trajectory prediction method provided in an exemplary embodiment of this application is shown. The viewport trajectory prediction method provided in this embodiment is applied to electronic devices, such as... Figure 2 As shown, the viewport trajectory prediction method may include, but is not limited to, steps S501 to S503:

[0128] Step S501: Obtain the point cloud video data to be detected;

[0129] Step S502: Input the point cloud video data to be detected into the viewport trajectory prediction model, and output a set of predicted viewport trajectories from the viewport trajectory prediction model. The set of predicted viewport trajectories includes multiple viewport trajectories predicted by the viewport trajectory prediction model. The viewport trajectory prediction model is obtained by the viewport trajectory prediction model training method described in any of the above embodiments.

[0130] Step S503: Filter the multiple viewport trajectories in the predicted viewport trajectory set to obtain the target viewport trajectory set.

[0131] In this implementation, after the point cloud video data to be detected is input into the viewport trajectory prediction model, the viewport trajectory prediction model outputs a set of predicted viewport trajectories. The set of predicted viewport trajectories includes multiple viewport trajectories predicted by the viewport trajectory prediction model, as well as the probability corresponding to each viewport trajectory. By filtering the multiple viewport trajectories in the set of predicted viewport trajectories, the target viewport trajectory set is obtained.

[0132] In this implementation, multiple possible viewport trajectories of the point cloud video data to be predicted are accurately predicted by the viewport trajectory prediction model, a set of predicted viewport trajectories is generated, and the viewport trajectories in the set of predicted viewport trajectories are filtered to obtain the set of target viewport trajectories, thereby further improving the accuracy of the predicted viewport trajectories.

[0133] In some implementations, the predicted viewport trajectory set also includes the probability corresponding to each viewport trajectory. The process of filtering multiple viewport trajectories in the predicted viewport trajectory set in step S503 to obtain the target viewport trajectory set may include, but is not limited to, steps S601 to S603.

[0134] Step S601: Calculate the entropy of the predicted viewport trajectory set based on the probability corresponding to each viewport trajectory in the predicted viewport trajectory set.

[0135] Step S602: Determine the number of viewport trajectories contained in the target trajectory set based on the entropy of the predicted viewport trajectory set.

[0136] Step S603: Select the number of viewport trajectories from the predicted viewport trajectory set to form the target viewport trajectory set.

[0137] In this implementation, entropy is used to measure the uncertainty of information. The higher the entropy of a set, the higher its uncertainty; conversely, the lower the entropy, the lower its uncertainty. In trajectory prediction, entropy can be used to measure the uncertainty of the probability distribution of each predicted trajectory.

[0138] Furthermore, by predicting the probability corresponding to each viewport trajectory in the viewport trajectory set, the entropy of the predicted viewport trajectory set is calculated. Specifically, it can be calculated using the following formula 5:

[0139] (5);

[0140] In the formula, Represents entropy, M This indicates the number of viewport trajectories in the predicted viewport trajectory set. i Indicates the first i Viewport trajectory, Indicates the first i The probability corresponding to each viewport trajectory.

[0141] Based on the entropy of the predicted viewport trajectory set, the number of viewport trajectories in the target trajectory set is determined by the entropy minimization method, and the number of viewport trajectories selected from the predicted viewport trajectory set is obtained.

[0142] In this implementation, the optimal number of predicted viewport trajectories is dynamically selected by minimizing entropy. This ensures high confidence in the prediction results while reducing uncertainty, thereby improving the accuracy of the predicted viewport trajectories. Furthermore, it avoids the problem of using a fixed threshold to select the number of trajectories, significantly reducing bandwidth usage.

[0143] In other implementations, the KL divergence method or Bayesian optimization method can be used to filter from the set of predicted viewport trajectories to obtain the set of target viewport trajectories.

[0144] Figure 3 This is a schematic diagram of the viewport trajectory prediction model training device provided in the embodiments of this application. Please refer to [link / reference]. Figure 3 This application also provides a viewport trajectory prediction model training device 700, which can implement the above-mentioned viewport trajectory prediction model training method. The viewport trajectory prediction model training device 700 includes:

[0145] The data acquisition module 701 is used to acquire labeled point cloud video data and unlabeled point cloud video data;

[0146] The first training module 702 is used to train the initial viewport trajectory prediction model using the labeled point cloud video data to obtain the first viewport trajectory prediction model.

[0147] The second training module 703 is used to train the first viewport trajectory prediction model using the unlabeled point cloud video data and preset spatiotemporal constraints to obtain a second viewport trajectory prediction model. The preset spatiotemporal constraints include at least one of a gaze-following loss condition, a smooth motion loss condition, and a context-dependent loss condition. The gaze-following loss condition is used to adjust the difference between the predicted points of the first viewport trajectory prediction model and the real points. The smooth motion loss condition is used to indicate the spatial coherence of the predicted points of the first viewport trajectory prediction model. The context-dependent loss condition is used to indicate the temporal consistency of the predicted points of the first viewport trajectory prediction model.

[0148] The third training module 704 is configured to, if the second viewport trajectory prediction model does not meet the preset convergence condition, use the second viewport trajectory prediction model as the initial viewport trajectory prediction model and re-execute the step of training the initial viewport trajectory prediction model with the labeled point cloud video data to obtain the first viewport trajectory prediction model; and re-execute the step of training the first viewport trajectory prediction model with the unlabeled point cloud video data and preset spatiotemporal constraints to obtain the second viewport trajectory prediction model, until the obtained second viewport trajectory prediction model meets the preset convergence condition, and use the second viewport trajectory prediction model as the trained viewport trajectory prediction model.

[0149] In some implementations, the preset constraints include the gaze-following loss condition, and the second training module 703 includes:

[0150] The first extraction submodule is used to extract key features from each video frame in the unlabeled point cloud video data to obtain multiple key features corresponding to each video frame.

[0151] The first execution submodule is used to execute the following for each video frame:

[0152] Each of the key features is calculated to obtain an attention score corresponding to each key feature; based on the attention scores corresponding to the key features, the predicted attention distribution of the video frame is obtained.

[0153] The first calculation submodule is used to calculate the gaze following loss value based on the predicted attention distribution corresponding to each video frame and the gaze point following loss condition.

[0154] The first training submodule is used to train the first viewport trajectory prediction model based on the gaze-following loss value to obtain the second viewport trajectory prediction model.

[0155] In some implementations, the preset constraints include the smooth motion loss condition, and the second training module 703 further includes:

[0156] The first acquisition submodule is used to acquire the predicted position of each video frame in the unlabeled point cloud video data by the first viewport trajectory prediction model.

[0157] The second execution submodule is used to perform the following for each video frame:

[0158] The speed change value of the video frame is calculated based on the predicted position of the video frame, the predicted position of the preceding video frame, and the predicted position of the second preceding video frame.

[0159] The second calculation submodule is used to calculate the smooth motion loss value based on the speed change value of each video frame and the smooth motion loss condition.

[0160] The second training submodule is used to train the first viewport trajectory prediction model based on the smooth motion loss value to obtain the second viewport trajectory prediction model.

[0161] In some implementations, the preset constraints include the context-dependent loss condition, and the second training module 703 further includes:

[0162] The second acquisition submodule is used to acquire the predicted position of each video frame in the unlabeled point cloud video data by the first viewport trajectory prediction model.

[0163] The second extraction submodule is used to extract context features from each video frame in the unlabeled point cloud video data to obtain the context features of each video frame.

[0164] The third execution submodule is used to perform the following for each video frame:

[0165] The context-dependent loss value of the video frame is calculated based on the predicted position of the next video frame and the context features of the video frame.

[0166] The third calculation submodule is used to calculate the context dependency loss value of the unlabeled point cloud video data based on the context dependency loss value of each video frame and the context dependency loss condition.

[0167] The third training submodule is used to train the first viewport trajectory prediction model based on the context dependency loss value of the unlabeled point cloud video data to obtain the second viewport trajectory prediction model.

[0168] The specific implementation of the viewport trajectory prediction model training device 700 is basically the same as the specific implementation of the viewport trajectory prediction model training method described above, and will not be repeated here.

[0169] Figure 4 This is a schematic diagram of the viewport trajectory prediction device provided in the embodiments of this application. Please refer to it. Figure 4 This application embodiment also provides a viewport trajectory prediction device 800, which can implement the above-described viewport trajectory prediction method. The viewport trajectory prediction device 800 includes:

[0170] The point cloud acquisition module 801 is used to acquire the point cloud video data to be detected.

[0171] The trajectory prediction module 802 is used to input the point cloud video data to be detected into the viewport trajectory prediction model, and the viewport trajectory prediction model outputs a set of predicted viewport trajectories. The set of predicted viewport trajectories includes multiple viewport trajectories predicted by the viewport trajectory prediction model. The viewport trajectory prediction model is obtained by any of the viewport trajectory model training methods in the above embodiments.

[0172] The trajectory filtering module 803 is used to filter multiple viewport trajectories in the predicted viewport trajectory set to obtain the target viewport trajectory set.

[0173] In some embodiments, the predicted viewport trajectory set further includes the probability corresponding to each viewport trajectory, and the trajectory filtering module 803 includes:

[0174] The entropy calculation submodule is used to calculate the entropy of the predicted viewport trajectory set based on the probability corresponding to each viewport trajectory in the predicted viewport trajectory set.

[0175] The quantity determination submodule is used to determine the number of viewport trajectories contained in the target trajectory set based on the entropy of the predicted viewport trajectory set.

[0176] The trajectory filtering submodule is used to filter the number of viewport trajectories from the predicted viewport trajectory set to form the target viewport trajectory set.

[0177] The specific implementation of the viewport trajectory prediction device 800 is basically the same as the specific embodiment of the viewport trajectory prediction method described above, and will not be repeated here.

[0178] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described viewport trajectory prediction model training method and / or viewport trajectory prediction method. This electronic device can be any smart terminal, including desktop computers, tablets, mobile phones, and in-vehicle computers.

[0179] Please see Figure 5 , Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device includes:

[0180] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0181] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 to execute the viewport trajectory prediction model training method and / or viewport trajectory prediction method of the embodiments of this application.

[0182] The input / output interface 903 is used to implement information input and output;

[0183] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0184] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0185] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0186] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described viewport trajectory prediction model training method and / or viewport trajectory prediction method.

[0187] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0188] In addition, this application embodiment also provides a computer program product for implementation, wherein when the instructions in the computer program product are executed by the processor of an electronic device, the electronic device implements the viewport trajectory prediction model training method and / or viewport trajectory prediction method in the above embodiments.

[0189] The viewport trajectory prediction model training method, viewport trajectory prediction method, and apparatus provided in this application training method use labeled point cloud video data to train an initial viewport trajectory prediction model, enabling the initial viewport trajectory prediction model to learn basic viewport trajectory prediction rules, resulting in a first viewport trajectory prediction model. Then, the first viewport trajectory prediction model is trained using unlabeled point cloud video data, with preset spatiotemporal constraints added during training to make the prediction results of the first viewport trajectory prediction model more consistent with natural spatiotemporal rules, resulting in a second viewport trajectory prediction model. If the second viewport trajectory prediction model does not meet preset convergence conditions, the above training process is repeated. By alternating between labeled and unlabeled data in each training iteration, the trained viewport trajectory prediction model can not only accurately predict the viewport trajectory of labeled point cloud video data but also extract useful information from unlabeled data to predict the viewport trajectory of unlabeled point cloud video data. This results in the trained viewport trajectory prediction model having better generalization ability and robustness, thus improving the adaptability of the viewport trajectory prediction model.

[0190] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0191] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0192] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0193] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0194] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0195] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0196] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0197] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0198] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0199] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0200] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for training a viewport trajectory prediction model, characterized in that, include: Obtain labeled point cloud video data and unlabeled point cloud video data; The initial viewport trajectory prediction model is trained using the labeled point cloud video data to obtain the first viewport trajectory prediction model. The first viewport trajectory prediction model is trained using the unlabeled point cloud video data and preset spatiotemporal constraints to obtain the second viewport trajectory prediction model. The preset spatiotemporal constraints include at least one of a gaze-following loss condition, a smooth motion loss condition, and a context-dependent loss condition. The gaze-following loss condition is used to adjust the difference between the predicted points of the first viewport trajectory prediction model and the real points. The smooth motion loss condition is used to indicate the spatial coherence of the predicted points of the first viewport trajectory prediction model. The context-dependent loss condition is used to indicate the temporal consistency of the predicted points of the first viewport trajectory prediction model. If the second viewport trajectory prediction model does not meet the preset convergence condition, then the second viewport trajectory prediction model is used as the initial viewport trajectory prediction model, and the step of training the initial viewport trajectory prediction model with the labeled point cloud video data to obtain the first viewport trajectory prediction model is re-executed. In addition, the step of re-exercising the first viewport trajectory prediction model with the unlabeled point cloud video data and preset spatiotemporal constraints to obtain the second viewport trajectory prediction model is repeated until the obtained second viewport trajectory prediction model satisfies the preset convergence condition, and the second viewport trajectory prediction model is used as the trained viewport trajectory prediction model.

2. The method according to claim 1, characterized in that, The preset constraints include the gaze point following loss condition; The step of training the first viewport trajectory prediction model using the unlabeled point cloud video data and preset spatiotemporal constraints to obtain the second viewport trajectory prediction model includes: Key features are extracted from each video frame in the unlabeled point cloud video data to obtain multiple key features corresponding to each video frame; For each video frame, perform: Each of the key features is calculated to obtain an attention score corresponding to each key feature; based on the attention scores corresponding to the key features, the predicted attention distribution of the video frame is obtained. The gaze following loss value is calculated based on the predicted attention distribution corresponding to each video frame and the gaze point following loss condition; The first viewport trajectory prediction model is trained based on the gaze-following loss value to obtain the second viewport trajectory prediction model.

3. The method according to claim 1, characterized in that, The preset constraints include the smooth motion loss condition; The step of training the first viewport trajectory prediction model using the unlabeled point cloud video data and preset spatiotemporal constraints to obtain the second viewport trajectory prediction model includes: Obtain the predicted position of each video frame in the unlabeled point cloud video data by the first viewport trajectory prediction model; For each video frame, execute: The speed change value of the video frame is calculated based on the predicted position of the video frame, the predicted position of the preceding video frame, and the predicted position of the second preceding video frame. The smooth motion loss value is calculated based on the speed change value of each video frame and the smooth motion loss condition. The first viewport trajectory prediction model is trained based on the smooth motion loss value to obtain the second viewport trajectory prediction model.

4. The method according to claim 1, characterized in that, The preset constraints include the context-dependent loss condition; The step of training the first viewport trajectory prediction model using the unlabeled point cloud video data and preset spatiotemporal constraints to obtain the second viewport trajectory prediction model includes: Obtain the predicted position of each video frame in the unlabeled point cloud video data by the first viewport trajectory prediction model; Context features are extracted from each video frame in the unlabeled point cloud video data to obtain the context features of each video frame; For each video frame, execute: The context-dependent loss value of the video frame is calculated based on the predicted position of the next video frame and the context features of the video frame. The context dependency loss value of the unlabeled point cloud video data is calculated based on the context dependency loss value of each video frame and the context dependency loss condition. The first viewport trajectory prediction model is trained based on the context dependency loss value of the unlabeled point cloud video data to obtain the second viewport trajectory prediction model.

5. A viewport trajectory prediction method, characterized in that, include: Acquire the point cloud video data to be inspected; The point cloud video data to be detected is input into the viewport trajectory prediction model, and the viewport trajectory prediction model outputs a set of predicted viewport trajectories. The set of predicted viewport trajectories includes multiple viewport trajectories predicted by the viewport trajectory prediction model. The viewport trajectory prediction model is obtained by the viewport trajectory prediction model training method according to any one of claims 1-4. The target viewport trajectory set is obtained by filtering multiple viewport trajectories in the predicted viewport trajectory set.

6. The method according to claim 5, characterized in that, The predicted viewport trajectory set also includes the probability corresponding to each viewport trajectory; The step of filtering multiple viewport trajectories in the predicted viewport trajectory set to obtain the target viewport trajectory set includes: The entropy of the predicted viewport trajectory set is calculated based on the probability corresponding to each viewport trajectory in the predicted viewport trajectory set. The number of viewport trajectories contained in the target viewport trajectory set is determined based on the entropy of the predicted viewport trajectory set. The target viewport trajectory set is formed by selecting the number of viewport trajectories from the predicted viewport trajectory set.

7. A viewport trajectory prediction model training device, characterized in that, The device includes: The data acquisition module is used to acquire labeled point cloud video data and unlabeled point cloud video data; The first training module is used to train the initial viewport trajectory prediction model using the labeled point cloud video data to obtain the first viewport trajectory prediction model. The second training module is used to train the first viewport trajectory prediction model using the unlabeled point cloud video data and preset spatiotemporal constraints to obtain a second viewport trajectory prediction model. The preset spatiotemporal constraints include at least one of a gaze-following loss condition, a smooth motion loss condition, and a context-dependent loss condition. The gaze-following loss condition is used to adjust the difference between the predicted points of the first viewport trajectory prediction model and the real points. The smooth motion loss condition is used to indicate the spatial coherence of the predicted points of the first viewport trajectory prediction model. The context-dependent loss condition is used to indicate the temporal consistency of the predicted points of the first viewport trajectory prediction model. The third training module is used to, if the second viewport trajectory prediction model does not meet the preset convergence condition, take the second viewport trajectory prediction model as the initial viewport trajectory prediction model and re-execute the step of training the initial viewport trajectory prediction model with the labeled point cloud video data to obtain the first viewport trajectory prediction model; and re-execute the step of training the first viewport trajectory prediction model with the unlabeled point cloud video data and preset spatiotemporal constraints to obtain the second viewport trajectory prediction model, until the obtained second viewport trajectory prediction model meets the preset convergence condition, and take the second viewport trajectory prediction model as the trained viewport trajectory prediction model.

8. A viewport trajectory prediction device, characterized in that, The device includes: The point cloud acquisition module is used to acquire the point cloud video data to be detected. The trajectory prediction module is used to input the point cloud video data to be detected into the viewport trajectory prediction model, and the viewport trajectory prediction model outputs a set of predicted viewport trajectories. The set of predicted viewport trajectories includes multiple viewport trajectories predicted by the viewport trajectory prediction model. The viewport trajectory prediction model is obtained by the viewport trajectory prediction model training method according to any one of claims 1-4. The trajectory filtering module is used to filter multiple viewport trajectories in the predicted viewport trajectory set to obtain the target viewport trajectory set.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the viewport trajectory prediction model training method as described in any one of claims 1 to 4, and / or the viewport trajectory prediction method as described in claims 5 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the viewport trajectory prediction model training method as described in any one of claims 1 to 4, and / or the viewport trajectory prediction method as described in claims 5 to 6.

Images