Holographic video adaptive transmission method based on QoE guarantee

Abstract

The invention discloses a QoE (Quality of Experience) guarantee-based holographic video adaptive transmission method, which comprises the following steps of: monitoring a network state in real time, and measuring and calculating key network parameters so as to evaluate a current network environment; the attention mechanism is combined with geometric features and color features to calculate the quality score of the point cloud video, meanwhile, an MANIQA model is introduced to extract depth features, noise and fuzzy features are fused, and the quality evaluation accuracy is improved; a deep learning method is adopted, the network parameters and the quality scores serve as input, a model capable of predicting subjective scores of the users is obtained through training, and a mapping relation from objective indexes to subjective experience is established; and dynamically adjusting a transmission strategy by combining the subjective score prediction result of the user based on the visual field range, bandwidth constraint and frame rate requirements of the user, and realizing efficient splicing and reconstruction of the point cloud video through visual field shielding level and quantization level control. The method has the advantages of cross-modal feature extraction and fusion and more accurate QoE prediction capability.

Description

Technical Field

[0001] This invention relates to the field of video transmission technology, and more specifically, to a holographic video adaptive transmission method based on QoE assurance. Background Technology

[0002] Currently, in traditional streaming media transmission research, QoS is related to bandwidth, latency, jitter, and throughput metrics, and how to quantify these metrics has been thoroughly studied. However, with the development of services and networks, the basis for allocating service network resources has shifted from being based on network parameters to being based on user experience. Therefore, a more comprehensive and multi-dimensional assessment of service quality is needed. Although spatial consistency (e.g., the realism of object shape, color, and texture) or temporal consistency (e.g., object motion) are major factors in perceived quality, their subjectivity is not easily measured. A typical process used to assess perceived quality is called QoE (Quality of Experience) estimation.

[0003] Over the past decade, Image Quality Assessment (IQA) and Video Quality Assessment (VQA) have been extensively studied to measure Quality of Experience (QoE). VQA assessments can generally be categorized into subjective and objective assessments. Subjective quality assessment is the most reliable method for evaluating perceived quality because video is geared towards the end user. Subjective quality is measured by asking a human subject to indicate the quality of an image or video, typically using a numerical scale such as the Mean Opinion Score (MOS) scale, which has five possible values ​​(1: Very Poor, 2: Poor, 3: Average, 4: Good, 5: Excellent). The statistical significance of the MOS value must be guaranteed. However, due to the rigor of the method, subjective assessment is very time-consuming, expensive, and impractical for real-time applications. Furthermore, subjective assessments can be influenced by many factors, making them impractical for many real-time applications, such as live video streaming. Meanwhile, objective evaluation metrics are widely used, such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Video Quality Metric (VQM). In general, subjective QoE assessment is extremely costly but highly accurate, while objective QoE assessment has lower accuracy but is far less difficult and time-consuming to collect data from. Therefore, much current research focuses on improving the accuracy of objective evaluation methods.

[0004] Objective quality assessment uses a computational VQA model that simulates HVS and human perception to automatically predict perceived video quality scores. The Video Quality Experts Group (VQEG) has conducted extensive research on the performance of objective VQA. Based on the availability of the original video, these assessments can be categorized into three types: Full-Reference (FR), Reduced-Reference (RR), and No-Reference (NR). In the FR method, the reference multimedia content is fully available for comparison with the received distorted content to assess visual quality. However, in most practical applications, the original signal is not available at the client or receiver end. In the RR method, we need to provide some features of the original image or video (i.e., texture, edges, etc.) for quality assessment, rather than providing the original image or video as a reference. RR is a practical method for real-time scenarios. No-Reference (NR) quality assessment in this category is performed blindly based on features extracted from the multimedia content being assessed because no reference is available. However, NR-based image and video quality assessment is a challenging task because the extracted features may provide very limited information.

[0005] Sam Van Damme et al. [1] studied the service quality degradation problem caused by factors such as bandwidth, latency and encoding during point cloud video streaming, and proposed an efficient no-reference (NR) objective quality assessment method. Traditional full-reference (FR) or reduced-reference (RR) indicators are not suitable for real-time applications due to the need for original undistorted content or complex calculations, while existing NR indicators are simple to calculate but have poor correlation with subjective perception. In order to solve these problems, a machine learning-based NR QoE assessment model was proposed. The model first uses the K-Nearest Neighbors (KNN) clustering mechanism to classify videos into specific categories, and then uses linear regression combined with sigmoid mapping to predict a set of NR metrics for each category. This method not only considers the diversity of video content, but also improves the accuracy of predicting user-perceived quality. Specifically, by calculating NR metrics including bandwidth, blur, noise, etc., and adjusting weights and parameters according to video category, the model can effectively estimate the quality of the video. The experimental part used an extended adaptive streaming media point cloud dataset, which contains 16 source videos and 453 sequences, covering a variety of scenarios and encoding conditions. The results show that the proposed model achieves a high PLCC (Pearson linear correlation coefficient) of 0.977 and a low RMSE (root mean square error) of 0.077 compared to the VMAF benchmark, significantly outperforming methods that do not use clustering or sigmoid mapping. Furthermore, the study found that adding a clustering step can improve model performance, especially when there are significant differences between different types of videos.

[0006] In summary, this work provides an efficient and accurate real-time quality assessment solution for point cloud video, suitable for dynamic adaptive streaming environments, and possesses high practical value and broad application prospects. Future research could further explore the applicability of this model at the GOP or frame level and examine its robustness to other types of compression distortion.

[0007] This method uses a set of objective NR metrics to classify a video sequence, pointing out that different types of videos can rely on completely different types of NR metrics for quality estimation.

[0008] The nonlinear relationship was not fully explored: Although linear regression combined with a sigmoid map was used to approximate the relationship between the NR metric and the quality score, this is still a relatively simplistic model assumption. In reality, there may be complex nonlinear relationships between the NR metric and subjectively perceived quality, and relying solely on linear models and sigmoid transformations may not be sufficient to capture all the nuances.

[0009] Lack of frame-level modeling: The paper primarily focuses on the average NR metric of the entire video segment. The averaging process may mask quality issues or instantaneous changes in certain keyframes, making it impossible to accurately identify local quality degradation. If the application scenario requires real-time feedback, segment-based methods may be less sensitive than frame-by-frame analysis. For applications requiring immediate response (such as real-time streaming monitoring), frame-by-frame analysis can provide faster feedback.

[0010] Wang et al. [2] studied the quality optimization problem in adaptive streaming of point clouds, especially the bit rate adaptive adjustment from the perspective of user experience quality (QoE). Traditional algorithms are mostly heuristic and lack worst-case performance guarantees. However, the QoE&Tile algorithm proposed by Wang et al. not only has theoretical performance guarantees, but also shows a near-optimal solution in practical applications, while significantly reducing computational complexity. This study first proposed a novel QoE metric through perspective projection analysis, projecting 3D tiles onto the display plane and measuring the contribution of each tile to QoE based on its 2D projection. For static point clouds, under a given bit budget, the bit rate adaptation problem of QoE optimization is formalized into a multiple choice knapsack problem, and further transformed into a submodulus function maximization problem. For dynamic point clouds that change over time, multiple voxel frames are combined into larger segments for transmission, and a segment-level QoE optimization bit rate adaptation problem is proposed. Both of these problems can be solved by the proposed efficient solution strategy based on a greedy algorithm. Extensive experiments across a range of static and dynamic scenarios, including static scenes with single or multiple point clouds from different viewpoints and dynamic scenes simulating a user's six-degree-of-freedom navigation path, validated the superiority of the proposed algorithm in terms of user visual quality and transmission efficiency. Results show that compared to existing solutions, the new algorithm excels in bandwidth utilization and approximation ratio, particularly in static scenes where the approximation ratio typically exceeds 99.88%, with a significantly reduced execution time. Furthermore, for dynamic point clouds, the algorithm effectively adjusts the transmission strategy according to changes in the user's viewpoint, ensuring an optimal user experience.

[0011] This method proposes a point cloud adaptive streaming framework driven by Quality of User Experience (QoE), which mainly solves the problem of efficient bit rate allocation for static and dynamic point clouds under different view conditions.

[0012] The method does not comprehensively consider the impact of network factors on QoE: It primarily focuses on improving Quality of User Experience (QoE) by optimizing point cloud partitioning strategies and bitrate adaptation, specifically considering factors such as view frustum, occlusion relationships, and rendering device resolution. However, it lacks a more comprehensive study of the influence of network factors on QoE. In practical applications, network conditions such as bandwidth fluctuations, latency, and packet loss rate significantly affect the user's viewing experience. Considering this, while the core focus of the method is on efficient content delivery optimization based on the user's viewport and content characteristics, a complete QoE optimization scheme typically requires more detailed consideration of network conditions.

[0013] Complexity and Personalized Experience: Traditional optimization algorithms may struggle to capture all the factors influencing QoE and their complex interactions. Deep learning models, with their powerful non-linear fitting capabilities, can learn the complex relationships between user preferences and content features from large-scale data, thereby providing more personalized QoE optimization solutions.

[0014] Real-time decision support: Traditional methods typically require solving complex optimization problems online, which can consume significant computational resources and introduce latency. In contrast, once trained, deep learning models can make predictions very quickly, which is particularly important for applications requiring low-latency responses.

[0015] Nguyen et al. [3] studied the impact of time quality changes and stuttering caused by unstable network conditions on user experience quality (QoE) during adaptive point cloud video streaming. First, a QoE database containing 248 test sequences was constructed. These sequences were generated based on four original point cloud videos and covered different types of time quality changes and stuttering modes. For time quality changes, multiple modes were designed, including constant quality, peak quality changes, single drop or rise; for stuttering, stagnation events of different lengths and frequencies were set. Subsequently, based on this database, the paper proposed two new QoE prediction models. The first model was used to quantify the impact of time quality changes on QoE. It predicted the user's QoE by calculating geometric QP, texture QP and bit rate, using a weighted summation method and using least squares to learn the model parameters to avoid overfitting. The second model focused on stuttering. It used gradient tree boosting method to predict based on factors such as total stuttering duration, number of stuttering events, minimum and maximum stuttering duration, and used decision tree ensemble to gradually update the model parameters. Experimental results show that both models can accurately predict subjective ratings and effectively quantify the impact of temporal quality variations and stuttering on user QoE. Furthermore, this research lays the foundation for further exploration of other factors that may affect QoE, such as initial latency, spatial quality variations, and user preferences. Plans are also underway to compare user QoE on different platforms, providing theoretical support and technical means for optimizing practical point cloud video streaming systems.

[0016] This method represents a significant advancement in studying the impact of temporal quality variations and stuttering on the quality of user experience (QoE) in adaptive point cloud video streaming.

[0017] Lack of spatial dimension consideration: This method primarily focuses on time-based chunking, evaluating QoE by analyzing quality variations and stuttering across different time periods. However, in practical applications, especially for 3D content such as point cloud videos, the spatial dimension is equally important. For example, as the user moves their viewpoint, certain areas may require higher resolution or finer detail processing. Simple time chunking cannot fully reflect this need.

[0018] The method fails to consider the impact of changes in user perspective on QoE. In applications such as virtual reality and augmented reality, the user's perspective is dynamic, and different perspectives may require different data quality and types. Ignoring this may result in a high overall QoE score, but with limited improvement in the experience for a specific user.

[0019] Single quality metric: This method uses geometry quantization parameter (GQP), texture quantization parameter (QP), and bit rate as the main indicators for evaluating point cloud quality. However, these indicators may not be sufficient to fully reflect the true quality of point cloud videos, especially in scenarios where high interactivity and immersive experiences are crucial. Summary of the Invention

[0020] The main objective of this invention is to provide a holographic video adaptive transmission method based on QoE assurance, so as to solve at least one technical problem in the prior art.

[0021] To achieve the above objectives, according to one aspect of the present invention, a holographic video adaptive transmission method based on QoE assurance is provided, comprising:

[0022] Real-time monitoring of network status, calculation of key network parameters such as jitter, latency, and available bandwidth to assess the current network environment;

[0023] The quality score of point cloud video is calculated by combining geometric and color features using an attention mechanism. At the same time, the MANIQA model is introduced to extract depth features and noise and blur features are fused to improve the accuracy of quality assessment.

[0024] By using deep learning methods, the network parameters and quality scores mentioned above are used as inputs to train a model that can predict users' subjective ratings, thus establishing a mapping relationship from objective indicators to subjective experience.

[0025] Based on the user's field of view, bandwidth constraints, and frame rate requirements, and combined with the above-mentioned user subjective rating prediction results, the transmission strategy is dynamically adjusted. By controlling the field of view occlusion level and quantization level, efficient stitching and reconstruction of point cloud video is achieved, so as to optimize the user experience and improve the utilization of network resources, and realize high-quality and smooth point cloud video transmission.

[0026] Preferably, the network parameters are obtained by analyzing the sending and receiving timestamps of periodic probe packets, thereby calculating network latency and jitter. At the same time, traffic monitoring technology is used to estimate the current available bandwidth, providing basic data for subsequent transmission decisions. This enables the system to dynamically adjust the transmission strategy according to the actual network conditions, avoiding video stuttering or image quality degradation caused by insufficient bandwidth or network fluctuations.

[0027] Preferably, through an attention mechanism, the system automatically identifies key areas that affect the visual experience and assigns them higher weights, thereby calculating quality scores more accurately, improving the accuracy of quality assessment, and providing reliable data support for subsequent user subjective rating predictions.

[0028] Preferably, a multilayer perceptron or long short-term memory network is used to capture the complex nonlinear relationships between different features and to consider the changing trends over time. During training, a large amount of user viewing experience data under different network conditions is first collected, including objectively measured network parameters, quality scores, and user-reported MOS values. Subsequently, supervised learning methods are used to train the model, and a cross-validation strategy is employed to optimize hyperparameters to ensure the model's generalization ability. After training, the model can predict possible subjective ratings from users based on real-time network conditions and video quality features, providing a basis for subsequent adaptive transmission strategies. This allows the system to dynamically adjust the transmission scheme under different network environments to maximize user experience.

[0029] Preferably, the system prioritizes transmitting point cloud data within the user's current field of view, while using lower quantization levels or direct occlusion for areas outside the field of view. This reduces unnecessary data transmission. During the stitching and reconstruction of point cloud video, the system uses multiple quantization levels to differentiate the encoding of point cloud data in different areas. High-priority areas use lower quantization steps to maintain higher visual quality, while low-priority areas use larger quantization steps to reduce bandwidth requirements. The system dynamically adjusts the quantization level selection based on real-time network bandwidth to ensure smooth video playback even with limited bandwidth. Finally, the quality score calculation module evaluates the quality of the reconstructed video, ensuring that the transmission strategy achieves efficient bandwidth utilization while guaranteeing user experience.

[0030] Preferably, in adaptive quality reconstruction, after dividing the point cloud into blocks, each point cloud is assigned a different quality level. First, an algorithm is used to determine the occlusion level of each point cloud block, identifying which point cloud blocks require high quality and which do not. Then, point cloud blocks with different quantization levels are traversed and allocated according to the level. The occlusion level represents the visibility of a point cloud block in the user's current field of view.

[0031] Preferably, the occlusion level is mainly based on the following factors: user's field of view direction, obtained through the head-mounted device or screen gaze point; angle between point cloud block and line of sight, calculated by the angle between the center of the point cloud block and the user's line of sight, the smaller the angle, the higher the priority; depth distance, the closer the point cloud block is to the user, the higher the priority; dynamic occlusion detection, using historical frame data or collision detection algorithms to determine whether it is occluded by other objects.

[0032] Preferably, point cloud blocks are mapped to corresponding quantization levels based on occlusion levels. Adaptive quality reconstruction includes: obtaining the center position of the point cloud block; calculating the angle between the viewing direction and the point cloud block direction; calculating the distance between the point cloud block and the user; determining whether it is occluded; comprehensively determining the occlusion level; mapping the occlusion level to the quantization level; and storing the results.

[0033] Preferably, the MLP model training and prediction steps include: data partitioning; data standardization; Bootstrap resampling; building the MLP model; compiling the model; model training; storing the model; model prediction; Bagging ensemble; calculating performance metrics; and returning the results.

[0034] Preferably, the point cloud block quality allocation steps include: initializing all blocks to the lowest quality; constructing occlusion level groups and sorting them from highest to lowest visibility; gradually reducing the occlusion level from the lowest occlusion level, iterating through the occlusion levels of each point cloud block; upgrading the quality level of the current block; backing up the current settings; calculating the total bandwidth consumption under the current configuration; if the required bit rate is greater than the available bandwidth; performing a rollback operation; if not, using an MLP model to predict the current QoE, predicting whether the QoE is less than the target QoE, and if the target QoE is not reached, continuing to attempt to upgrade, and exiting early after reaching the target; then, mapping IDs to quality level strings; and finally, evaluation.

[0035] The technical solution of this invention has the following technical effects:

[0036] Cross-modal feature extraction and fusion: Utilizing geometric features, color features, and projected image features, data fusion is performed through an attention mechanism to improve the accuracy of quality assessment. Multi-dimensional QoE modeling: Simultaneously considering network parameters (such as jitter, latency, and bandwidth) and content parameters (such as the geometric and color features of point clouds), a complex nonlinear relationship model is constructed to predict user subjective ratings. Dynamic resource allocation strategy: Based on the user's field of view, bandwidth limitations, and frame rate requirements, efficient point cloud video stitching and reconstruction are achieved, and transmission strategies are optimized to ensure user experience.

[0037] More accurate QoE prediction capability: By introducing deep learning models, more complex and detailed nonlinear mapping relationships can be captured, thereby improving the accuracy of QoE prediction. Comprehensive consideration of network factors: Not only focusing on the characteristics of the point cloud content itself, but also fully considering the impact of network state changes on user experience, achieving a more comprehensive service quality assessment. Personalized experience optimization: The transmission strategy can be dynamically adjusted according to different user behavior patterns and preferences, providing a more personalized viewing experience. Real-time response mechanism: With the help of pre-trained deep learning models, online computation requires almost no additional time overhead, ensuring low-latency response speed. Attached Figure Description

[0038] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0039] Figure 1 A no-reference quality model diagram of clustered point cloud video psychometrics based on the QoE-guaranteed holographic video adaptive transmission method according to the present invention is shown.

[0040] Figure 2 It shows Figure 1 A view of a tile-based QoE-driven adaptive streaming media method for holographic video adaptive transmission based on QoE guarantees;

[0041] Figure 3 It shows Figure 1 An adaptive point cloud video stream system view based on a QoE-guaranteed holographic video adaptive transmission method;

[0042] Figure 4 It shows Figure 1 A view of the content feature extraction model of the holographic video adaptive transmission method based on QoE assurance;

[0043] Figure 5 It shows Figure 1 A view of the MLP structure of the holographic video adaptive transmission method based on QoE guarantee;

[0044] Figure 6 It shows Figure 1 A time delay analysis view of the MTP components in the holographic video adaptive transmission method based on QoE guarantee;

[0045] Figure 7 It shows Figure 1 Parallel projection and perspective projection views of a holographic video adaptive transmission method based on QoE guarantees;

[0046] Figure 8 It shows Figure 1 The flowchart of the holographic video adaptive transmission method based on QoE guarantee, which utilizes perspective projection to realize the transformation of three-dimensional coordinates to two-dimensional coordinates;

[0047] Figure 9 It shows Figure 1 A perspective projection of a frustum view in a holographic video adaptive transmission method based on QoE guarantees;

[0048] Figure 10 It shows Figure 1 The viewport transformation from NDC space to screen space in the holographic video adaptive transmission method based on QoE guarantee. Detailed Implementation

[0049] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0050] like Figures 1 to 10 As shown, this embodiment of the invention provides a holographic video adaptive transmission method based on QoE assurance, including: real-time monitoring of network status, calculating key network parameters such as jitter, latency, and available bandwidth to assess the current network environment; calculating the quality score of point cloud video using an attention mechanism combined with geometric and color features, while introducing the MANIQA model to extract depth features and fusing noise and blur features to improve the accuracy of quality assessment; using a deep learning method, taking the above network parameters and quality score as input, training a model that can predict user subjective ratings, and establishing a mapping relationship from objective indicators to subjective experience; dynamically adjusting the transmission strategy based on the user's field of view, bandwidth constraints, and frame rate requirements, combined with the above user subjective rating prediction results, and achieving efficient stitching and reconstruction of point cloud video through field of view occlusion level and quantization level control, so as to optimize user experience and improve network resource utilization, and achieve high-quality and smooth point cloud video transmission.

[0051] In this embodiment, the network parameters are obtained by analyzing the sending and receiving timestamps of periodic probe packets, thereby calculating network latency and jitter. At the same time, traffic monitoring technology is used to estimate the current available bandwidth, providing basic data for subsequent transmission decisions. This enables the system to dynamically adjust the transmission strategy according to the actual network conditions, avoiding video stuttering or image quality degradation caused by insufficient bandwidth or network fluctuations.

[0052] In this embodiment, through the attention mechanism, the system automatically identifies key areas that affect the visual experience and assigns them higher weights, thereby calculating the quality score more accurately, improving the accuracy of quality assessment, and providing reliable data support for subsequent user subjective rating prediction.

[0053] In this embodiment, a multilayer perceptron or long short-term memory network is used to capture the complex nonlinear relationships between different features and to consider the changing trends over time. During training, a large amount of user viewing experience data under different network conditions is first collected, including objectively measured network parameters, quality scores, and user-reported MOS values. Subsequently, supervised learning methods are used to train the model, and a cross-validation strategy is employed to optimize hyperparameters to ensure the model's generalization ability. After training, the model can predict possible subjective ratings from users based on real-time network conditions and video quality features, providing a basis for subsequent adaptive transmission strategies. This allows the system to dynamically adjust the transmission scheme under different network environments to maximize user experience.

[0054] In this embodiment, the system prioritizes transmitting point cloud data within the user's current field of view, while using lower quantization levels or direct occlusion for areas outside the field of view. This reduces unnecessary data transmission. During the stitching and reconstruction of point cloud video, the system uses multiple quantization levels to differentiate the encoding of point cloud data in different regions. High-priority regions use lower quantization steps to maintain higher visual quality, while low-priority regions use larger quantization steps to reduce bandwidth requirements. The system dynamically adjusts the quantization level selection based on real-time network bandwidth to ensure smooth video playback even with limited bandwidth. Finally, the quality score calculation module evaluates the quality of the reconstructed video, ensuring that the transmission strategy achieves efficient bandwidth utilization while guaranteeing user experience.

[0055] In this embodiment, adaptive quality reconstruction divides the point cloud into blocks, and each point cloud is assigned a different quality level. First, an algorithm is used to determine the occlusion level of each point cloud block to identify which point cloud blocks require high quality and which do not. Then, point cloud blocks with different quantization levels are traversed and allocated according to the level. The occlusion level represents the visibility of a point cloud block in the user's current field of view.

[0056] In this embodiment, the occlusion level is mainly based on the following factors: user's field of view direction, obtained through head-mounted devices or screen gaze points; angle between point cloud blocks and line of sight, calculated by the angle between the center of the point cloud block and the user's line of sight, the smaller the angle, the higher the priority; depth distance, the closer the point cloud block is to the user, the higher the priority; dynamic occlusion detection, using historical frame data or collision detection algorithms to determine whether it is occluded by other objects.

[0057] In this embodiment, point cloud blocks are mapped to corresponding quantization levels according to the occlusion level. The adaptive quality reconstruction includes: obtaining the center position of the point cloud block; calculating the angle between the viewing direction and the direction of the point cloud block; calculating the distance between the point cloud block and the user; determining whether it is occluded; comprehensively judging the occlusion level; mapping the occlusion level to the quantization level; and storing the results.

[0058] In this embodiment, the MLP model training and prediction steps include: data partitioning; data standardization; Bootstrap resampling; building the MLP model; compiling the model; model training; storing the model; model prediction; Bagging ensemble; calculating performance metrics; and returning the results.

[0059] In this embodiment, the point cloud block quality allocation steps include: initializing all blocks to the lowest quality; constructing occlusion level groups and sorting them from highest to lowest visibility; gradually reducing the occlusion level from the lowest occlusion level, iterating through the occlusion levels of each point cloud block; upgrading the quality level of the current block; backing up the current settings; calculating the total bandwidth consumption under the current configuration; if the required bit rate is greater than the available bandwidth; performing a rollback operation; if not, using an MLP model to predict the current QoE, predicting whether the QoE is less than the target QoE, and if the target QoE is not reached, continuing to attempt to upgrade, and exiting early after reaching the target; then, mapping IDs to quality level strings; and finally, evaluation.

[0060] In this embodiment, a method for evaluating the quality of point cloud content under different viewports is designed. Based on the geometric and color features of the 3D model, and a projection-based deep learning network, a cross-modal attention fusion mechanism is used to quantify the quality evaluation scores of point clouds under different viewports. A QoE learning model for network parameters and content parameters is also considered. The correlation between various influencing factors is explored, and the mapping curves between each factor and QoE are fitted, laying the foundation for resource allocation for adaptive transmission. Different spatial partitioning strategies and downsampling levels are used. Based on the established model, point cloud reconstruction and dynamic rendering are achieved based on network conditions and user behavior. The aim is to achieve adaptive dynamic transmission of point cloud video to optimize user experience and improve network resource utilization. The overall system architecture consists of four main parts, involving network parameter measurement, quality score calculation, user subjective rating prediction, and dynamic transmission strategies based on visual perception and bandwidth limitations.

[0061] In this embodiment, the first part involves real-time monitoring of the network status, including the calculation of key network parameters such as jitter, latency, and available bandwidth. This part ensures that the system can accurately assess the current network environment and adjust the transmission strategy accordingly to avoid video stuttering or image quality degradation caused by network fluctuations. The second part uses an attention mechanism combined with geometric and color features to calculate the quality score, while introducing the MANIQA model to extract deep features and fusing noise and blur features to improve the accuracy of quality assessment. The third part employs deep learning methods, using network parameters and the quality score as input to train a model that can predict user subjective ratings (MOS), thereby establishing a mapping relationship from objective indicators to subjective experience. The fourth part dynamically adjusts the transmission strategy based on the user's field of view, bandwidth constraints, and frame rate requirements, achieving efficient stitching and reconstruction of point cloud video through field of view occlusion and quantization level control, ultimately achieving high-quality and smooth point cloud video transmission.

[0062] In this embodiment, network parameter calculation and quality assessment are crucial because network conditions directly impact user experience during point cloud video transmission. Therefore, the core task of the first part is to accurately calculate network parameters, including jitter, latency, and available bandwidth. These parameters are obtained through analysis of the sending and receiving timestamps of periodic probe packets, thereby calculating network latency and jitter, while simultaneously estimating the current available bandwidth using traffic monitoring technology. The output of this module provides foundational data for subsequent transmission decisions, enabling the system to dynamically adjust transmission strategies based on actual network conditions, avoiding video stuttering or image quality degradation due to insufficient bandwidth or network fluctuations. The second part, the quality assessment module, is based on an attention mechanism, combining geometric features (such as spatial distribution statistics of point clouds) and color features (such as LAB color distribution) to measure video quality. Furthermore, this module utilizes the MANIQA network to extract high-dimensional features and fuses noise and blur features to enhance the robustness of the quality assessment. Through the attention mechanism, the system can automatically identify key areas affecting the visual experience and assign them higher weights, thereby calculating quality scores more accurately. This process not only improves the accuracy of quality assessment but also provides reliable data support for subsequent user subjective rating predictions.

[0063] In this embodiment, the core objective of the third part—user subjective rating prediction and model training—is to establish a mapping relationship from network features and quality scores to user subjective ratings (Mean Opinion Score, MOS). To this end, the system employs a deep learning model, using network parameters (such as jitter, latency, and bandwidth), frame rate, and the quality score calculated in the second part as input, and user-generated real rating data as labels for training. This model typically uses a Multilayer Perceptron (MLP) or Long Short-Term Memory (LSTM) network to capture the complex nonlinear relationships between different features and consider time-series trends. During training, a large amount of user viewing experience data under different network conditions is first collected, including objectively measured network parameters, quality scores, and user-reported MOS values. Subsequently, supervised learning methods are used to train the model, and a cross-validation strategy is employed to optimize hyperparameters to ensure the model's generalization ability. After training, the model can predict possible user subjective ratings based on real-time network conditions and video quality features, thus providing a basis for subsequent adaptive transmission strategies. This process enables the system to dynamically adjust the transmission scheme under different network environments to maximize user experience.

[0064] In this embodiment, the core of the adaptive dynamic transmission strategy, Part Four, lies in dynamically adjusting the transmission strategy of point cloud video based on user perspective, network bandwidth limitations, and frame rate requirements. This module integrates the outputs of the first three parts, including network parameters, quality scores, and user subjective rating prediction models, to determine the optimal transmission quality. Specifically, the system prioritizes transmitting point cloud data within the user's current field of view, while using lower quantization levels or direct occlusion for areas outside the field of view, thereby reducing unnecessary data transmission. During the stitching and reconstruction of point cloud video, the system uses multiple quantization levels (four in total) to differentially encode point cloud data in different regions. High-priority areas use lower quantization steps to maintain higher visual quality, while low-priority areas use larger quantization steps to reduce transmission bandwidth requirements. Furthermore, the system dynamically adjusts the selection of quantization levels based on real-time network bandwidth to ensure smooth video playback even with limited bandwidth. Finally, the quality score calculation module evaluates the quality of the reconstructed video, ensuring that the transmission strategy achieves efficient bandwidth utilization while guaranteeing user experience.

[0065] The relevant mathematical model for this embodiment is as follows:

[0066] Network parameter modeling

[0067] ① Packet Loss: The number of data packets lost along the path from source to destination on the network, which directly affects the completeness of the model as perceived by the user.

[0068] Packet loss rate = (Total packets sent - Total packets received) / Total packets sent * 100%.

[0069] ② Delay is the time required for a specified data packet to be transmitted from the source to the destination.

[0070]

[0071] In the formula: The request latency, transmission latency, processing latency, queuing latency, and in-band allocation latency are respectively divided into n regions within the throughput area, representing traffic U. j And for each interval, the flow rate U ij Response delay T ji The probability p i Calculations are performed, therefore, the network's balanced transmission delay ET is calculated using the formula. j :

[0072]

[0073] Among them, ET j For network-balanced transmission delay; T jiFlow U in each interval ij Corresponding delay; p i Flow U in each interval ij Response delay T ji The probability of n; n represents n regions.

[0074] ③ Jitter: The difference in one-way delay when transmitting data packets over a network. Voice and video communications are particularly susceptible to jitter. In holographic communication, due to the large amount of data per frame, jitter has a more pronounced impact on the continuity of viewing; the standard deviation method is used for calculation.

[0075]

[0076] Where n is the number of reported delays, based on the n reported delays D1, D2, ..., D... n The standard deviation is calculated based on the sampling; the smaller the standard deviation, the smaller the jitter. S is the standard deviation of the delay jitter, and D1, D2, ..., D... n It is the time delay from sending the diagnostic packet to successfully receiving the corresponding diagnostic response packet. It is the average time delay.

[0077] ④MTP is the motion-to-image delay, which is the time from when the user moves to when the system adjusts the viewing angle based on this movement and displays it on the device. This directly affects the occurrence of motion sickness. In real-time communication, time constraints are needed for the entire process from data generation to display. To accurately measure MTP delay, this study defines MTP delay in volumetric video streaming systems as follows:

[0078] T mtp =T server +T network +T client (4-5)

[0079] T server ,T network and T client The definition is as follows:

[0080] T server =T rend +T enc (4-6)

[0081] T client =T dec +T disp (4-7)

[0082] Among them, T server This refers to the latency occurring on the server side, where T... rend T refers to the time required to render the volume image corresponding to the user's current viewport; encThis refers to the time required to convert the 3D view of a rendered stereoscopic video into a 2D image and then encode it using a video codec, such as H.264 / AVC or H.265 / HEVC. network This refers to the total latency that occurs during server-client data communication, where T... up and T down This represents the propagation delay between the server and the client; and T trans This refers to the time required to transmit data. T client This refers to the client's latency, where T... dec It is the time required to decode the received video data; and T disp This refers to the delay time for outputting the decoded image on the screen.

[0083] ⑤ Bandwidth refers to the available network transmission bandwidth at the current time. On the main transmission path, passive measurement technology is used. If at time t... k At time 1, a SACK (Selective Acknowledgment) packet arrives at the sender, and the total length of the acknowledgment data packet is d. k Theoretically speaking, t k The real-time available bandwidth at any given time can be determined by the following formula:

[0084]

[0085] Among them, t k-1 It is the reception time of the previous SACK; while △t k This is the time interval between the arrival of two consecutive SACK packets.

[0086] Content parameter modeling:

[0087] First, statistical features are extracted from the 3D point cloud. Second, the 3D point cloud is projected onto a 2D plane, and features are extracted from these 2D images. Finally, a data fusion model is used to combine statistical and image features, which together constitute a quality assessment of the 3D point cloud.

[0088] Model-based feature extraction is primarily used to extract geometric, color, and statistical features. This project employs a statistical feature extraction model based on the concept of Natural Scene Statistics (NSS) to extract 3D statistical features. These statistical features are calculated from geometric and color features. The color map provides texture details about the point cloud, while the depth map provides geometric information about the shape.

[0089] ① Geometric Features: The first step is to use the k-nearest neighbor algorithm to determine the local neighborhood of each point in the point cloud, where the neighborhood of a given point consists of the nearest k points based on Euclidean distance. Once the neighborhood is defined, for each point p... i Calculate the covariance matrix C using the following formula. i :

[0090]

[0091] Where, p j It is p i Points within the neighborhood, where K is the number of points, ( It is the centroid. This matrix is ​​relative to p. i The spatial distribution of surrounding points is measured, capturing the degree of variation of these points along different coordinate axes. Then, the eigenvalues ​​and eigenvectors of the covariance matrix are calculated. The eigenvalues ​​λ... x ,λ y ,λ z It represents the variance of points along the coordinate axes as defined by their corresponding eigenvectors, and helps derive five key geometric features:

[0092] • Curvature: Measured as It indicates the degree to which a curved surface deviates from flatness and can be used as a unit to describe roughness or smoothness.

[0093] • Anisotropy: It describes the variance geometrically along different directions.

[0094] • Linearity: It assesses the alignment of a point with a line.

[0095] • Planarity: It evaluates how well points fit into a plane.

[0096] · Sphericity: It indicates the degree to which points form a sphere.

[0097] ② Color Features: Color features play a crucial role in evaluating the visual quality of 3D models. In a color point cloud, each point is represented by color information typically encoded in the RGB color space. However, the RGB color space does not correlate well with human color perception. To address this issue, we convert RGB color values ​​to the CIELAB (LAB) color space, which better aligns with human visual sensitivity to color. It is designed to achieve a nearly uniform interval between perceived color differences. The LAB color space represents color as three values:

[0098] ·L: Lightness from black (0) to white (100).

[0099] • A: A position between red and green, technically an unbounded value, but usually kept within the range of integers.

[0100] • B: Positions between yellow and blue, technically unbounded, but kept within an integer range.

[0101] This conversion involves first transforming the RGB values ​​to the XYZ color space, which acts as a bridge to LAB. The XYZ to LAB conversion aims to mimic the non-linear response of the eye, emphasizing the scaling of color differences based on their perceived importance. The full mathematical explanation of this transformation is as follows:

[0102]

[0103] Where R, G, B represent the corresponding RGB color channels, X, Y, Z represent the corresponding XYZ color channels, and L, A, B represent the corresponding RGB color channels. n ,Y n Z n Describes the specified white achromatic reference light source, with δ set to Finally, the LAB color channels are calculated as color feature domains.

[0104] Statistical Feature Estimation: Based on the feature domains obtained above, statistical feature estimation is performed. Visual quality can be comprehensively assessed based on changes caused by different types of distortion. The following parameters of the NSS model are significantly affected by distortion and are suitable for quality assessment. We selected statistical parameters such as mean (μ), standard deviation (σ), entropy (H), generalized Gaussian distribution (GGD), asymmetric generalized Gaussian distribution (AGGD), and Gamma parameter (γ). Based on this, for the geometric domain, we obtained 11 statistical parameters: μ, σ, H, GGD(shape,σ), AGGD(asymmetry coefficient,shape,scale), and γ(shape,scale). For the color gamut, we have 3 statistical parameters: μ, σ, and H.

[0105] For each point cloud, a total of 64 features were calculated. This means that there are 55 features (5×11) in the geometric domain (curvature, anisotropy, linearity, flatness, sphericity) and 9 features (L, a, b) in the color domain (3×3). Finally, normalization was performed using a Min-Max scaler based on all NSS features to obtain a single feature vector for a given point cloud.

[0106]

[0107] Among them, F NSS It is an NSS feature belonging to a 64×1 dimensional space.

[0108] The feature extraction for the second projection method mainly consists of two main modules: (i) the projection module converts the 3D point cloud into a series of 2D images, and (ii) the deep learning module extracts features from these images.

[0109] ③ Projection Module: Since the final reconstructed point cloud is presented on the display screen based on the user's perspective, we considered projecting each 3D tile onto a 2D display plane to measure its effectiveness. Because the projection is on a plane rather than a curved surface, objects at greater distances appear smaller than those closer in the perspective-projected image, which aligns with the visual effects of photographic systems and human vision. Perspective projection displays a more realistic appearance than parallel projection and was adopted in the project.

[0110] The flowchart illustrates the transformation from 3D geometric coordinates to 2D coordinates on the display screen. The geometric coordinates of the original 3D point cloud are first transformed from object space to eye space by multiplying with the Model-View matrix. Then, perspective projection is used to convert the coordinates in eye space to Normalized Device Coordinates (NDC). Finally, viewport transformation is performed on the NDC to obtain the 2D coordinates in screen space. Specifically, these transformation matrices are described below.

[0111] Model-view matrix: Let (x obj ,y obj ,z obj ,1) represents the geometric coordinates of a point in the object space of the original 3D point cloud, (x eye ,y eye ,z eye ,1) represents the coordinates in the eye-space, then we have

[0112]

[0113] Among them, M modelview It is M model and M view The product of M, where M model M is the model matrix that transforms coordinates from object space to world space. view It is a view matrix that transforms coordinates from world space to eye space.

[0114] Perspective projection: Figure 9In perspective projection, the view volume is a truncated pyramidal view cone, formed by a pair of truncated pyramids of parallel planes (z = -n and z = -f) perpendicular to the central axis. Points in the view cone represented in eye-space coordinates are first transformed to coordinates in clip space using a projection matrix. Then, perspective partitioning maps the clip space coordinates to the NDC, where the x, y, and z coordinates all range from [-1, 1]. Specifically, let (x... clip ,y clip ,z clip ,w clip () represents the coordinates within the clipping plane:

[0115]

[0116] Where M projection Let M represent the perspective projection matrix. If the view frustum is symmetric, and its central axis is set as the z-axis, then the M-projection can be expressed as:

[0117]

[0118] Where width and height represent the width and height of the near clipping plane of the view frustum, respectively. n and f are the z-coordinates of the near and far clipping planes of the view frustum. The projection matrix can be further simplified to...

[0119]

[0120] Here, the range of the vertical viewing angle is denoted as FOV (Field of View), and the aspect ratio of the near clipping plane is denoted as ratio. n and f are the z-coordinates of the near and far clipping planes of the view frustum.

[0121] The perspective projection matrix described above can be derived using linear transformations and the properties of similar triangles. To further convert the clipping coordinates to NDC, perspective partitioning needs to be applied, which can be written as...

[0122]

[0123] Where (x) ndc ,y ndc ,z ndc (x) represents the coordinates in the normalized device space. clip ,y clip ,z clip ,w clip ) represents the coordinates within the clipping plane.

[0124] To adapt to a 2D rendering screen, a viewport transformation is performed on the NDC to generate coordinates in screen space. Screen coordinates originate from (s... x ,s y ,0) changes to (s x +ws ,s y +h s The range of the z-coordinate in screen space is [n, 0), s ,f s Within ] . Among them, (x screen ,y screen ,z sreen () represents the coordinates in screen space, z screen It is fixed to 0 because it is on the screen, w s and h s These refer to the changes in the width and height coordinates (screen width and height), respectively.

[0125]

[0126] After viewport transformation, several points in screen space may share the same (x) screen ,y srcreen ), but with different z screen Of these, only z screen The smallest point It is visible because other points are... Occlusion. By selecting the set of visible points. We can obtain the 2D coordinates of the pixels rendered on the display screen, i.e., the visible point set. of (x) screen ,y screen ).

[0127] Therefore, the QoE of a point cloud should be determined by the user's view frustum and position, the display resolution, and the level of detail of the 3D model. Intuitively, if a 3D model is invisible (e.g., outside the user's view frustum or completely occluded by another model), then that model contributes nothing to the visual quality of the viewport on the display plane. On the other hand, for visible models, their QoE is generally considered to be inversely proportional to the distance from the model to the user, which is also constrained by the user's display resolution. Furthermore, the impact of model detail can be easily understood as higher data volume corresponds to higher visual quality, thus contributing more to the user's QoE. Considering all of the above, we define an overall QoE metric for a 3D tile Ti, whose representation quality level in the user's view frustum is q:

[0128] Q i,q =UPB i ·B i,q (4-21)

[0129] Among them, B i,q Let q represent the number of bits required to represent the q-th quality level of the encoded tile Ti, and UPBi represent the utility per bit of the model, which is...

[0130]

[0131] Where, N v,i N represents the number of voxels of tile i in the user's view frustum. p,i B represents the number of corresponding pixels projected onto the display screen via perspective projection. i,max N is a normalization parameter representing the number of bits required to encode tile i at the highest quality level. Formally, N... v,i and N p,i This can be determined using the symbols introduced earlier. Specifically, if we use p ~ (x obj ,y obj ,z pbj Let N represent the correspondence between point p and its original three-dimensional geometric coordinates in object space. v,i =|V i |, Where V i ={p~(x obj ,y obj ,z obj )}|p∈T i , (x ndc ,y ndc ,z ndc )∈[-1,1] 3}

[0132] To extract features from the P projection, the MANIQA network is used. For a specific point cloud, each P projection is processed by the modified MANIQA network to obtain P feature vectors:

[0133]

[0134] Where p∈1,2,…,n is the p-th projection of n projections, C I This represents the number of features extracted. Then, the element-wise average of the P image feature vectors is performed to obtain the result of a given length C. I A single feature vector F of the point cloud I .

[0135] In order to use multimodal feature vectors and To fully assess the quality of point clouds, a symmetric cross-modal attention mechanism was introduced.

[0136] First, a learnable linear transformation is used to linearly project the two sets of features into a common dimensional space:

[0137]

[0138] Among them W NSSand W I Let be the projection matrix. To adjust features.

[0139] The next stage involves a multi-head attention mechanism, enabling the model to simultaneously focus on the information components of two modalities. Using symmetric cross-modal attention, features from both modalities guide and complement each other to produce a final feature vector, which is derived from the original feature vector. and It is formed by concatenating with the multimodal feature vector obtained from the attention part:

[0140]

[0141] Where ⊕(·) indicates cascading, and Ψ(·) is the operation performed by the symmetric cross-modal attention module, producing attention-enhanced feature representations. It encapsulates rich information across modalities.

[0142] The relevant algorithm flow in this embodiment is as follows:

[0143] ① Adaptive quality reconstruction (pre-reconstruction): After we divide the point cloud into blocks, each point cloud will be assigned a different quality level. We first use a certain algorithm to determine the occlusion level of each point cloud block, to determine which point cloud blocks need high quality and which do not. Then, based on the level, we iterate and allocate point cloud blocks with different quantization levels.

[0144] Occlusion level indicates the visibility of a point cloud patch within the user's current field of view. It is primarily based on the following factors:

[0145] • User field of view (FoV Direction): Obtained through the head-mounted device or screen gaze point;

[0146] • Angle between point cloud block and view: Calculate the angle between the center of the point cloud block and the user's view. The smaller the angle, the higher the priority.

[0147] • Depth and Distance: Point cloud blocks that are closer to the user have higher priority;

[0148] • Dynamic occlusion detection: Uses historical frame data or collision detection algorithms to determine whether the object is occluded by other objects.

[0149] The occlusion level can be divided into 4 levels (0-3), where:

[0150] • Level 0 (Invisible): Completely blocked, no transmission required;

[0151] • Level 1 (low priority): Edge areas or long distances, compressible;

[0152] Level 2 (Medium Priority): Partially visible or at medium distance;

[0153] Level 3 (High Priority): Main field of view, close range, unobstructed.

[0154] The system offers four quantization levels (QL0 to QL3), each corresponding to different compression intensities and image quality losses:

[0155] Quantitative Grade Allocation Table

[0156] Quantitative Level Compressive strength quality loss Applicable Scenarios QL0 Extremely high great Invisible area QL1 high big Edge area QL2 middle medium Secondary areas QL3 Low Small Main field of view

[0157] Map point cloud blocks to the corresponding quantization level based on the occlusion level.

[0158] The following is the pseudocode implementation of the adaptive quality reconstruction module, which includes two core steps: occlusion level determination and quantization level assignment:

[0159]

[0160]

[0161] Calculation of the angle of sight:

[0162]

[0163] in, It is the user's current view direction vector. Let θ represent the vector pointing from the user's position to the center of the point cloud patch, and θ be the angle between the two vectors (in degrees). This formula is used to measure whether a point cloud patch is within the user's field of view (FoV). The smaller the angle, the closer the point cloud patch is to the user's gaze direction, and the higher its visual importance.

[0164] Euclidean distance between a point and a user:

[0165]

[0166] in, These are the center coordinates of the point cloud block. Here, is the user's current position, d is the Euclidean distance between them, and (x1, y1, z1) and (x2, y2, z2) are the coordinates of these two points, respectively. This formula is used to determine the distance between point cloud blocks and the user. Point cloud blocks that are closer to the user have a greater visual impact on the user and therefore should be assigned a higher quality level.

[0167] Obstruction level determination rules:

[0168]

[0169] Here, occlusion_level is the occlusion level, is_occluded is a boolean value indicating whether it is occluded, angle is the angle between the viewing direction and the point cloud block direction, and distance is the distance between the point cloud block and the user.

[0170] This logical rule divides point cloud blocks into four occlusion levels (0-3), representing invisible, edge-visible, partially visible, and visible in the main field of view, respectively. Visual priority is determined by combining two dimensions: angle and distance.

[0171] ②MLP model training algorithm:

[0172] Deep learning algorithms (such as multilayer perceptrons, MLPs) are well-suited for modeling the complex nonlinear relationship between QoS parameters and QoE scores. By training on a dataset containing network parameters and content features, a mapping model from objective parameters to subjective experience scores (such as MOS) can be constructed.

[0173]

[0174]

[0175] Hidden layer neuron computation (including activation functions):

[0176]

[0177] Among them, h mn It is the output of the nth neuron in the mth layer, where x is the input vector and ω is the output of the nth neuron in the mth layer. mn c is the weight vector of the neuron. m It is a bias term, which uses the ReLU activation function to introduce nonlinearity.

[0178] Output layer linear regression:

[0179]

[0180] Where y is the final output value (QoE prediction value), h Mi It is the output of the last hidden layer, ω Mi ,c M These are the weights and biases of the output layer.

[0181] Loss function – Mean Squared Error (MSE):

[0182]

[0183] Among them, y i This is the actual QoE value. This is the model's predicted value, and L is the total number of samples, used to optimize the model parameters (weights and biases).

[0184] Coefficient of determination R 2 (Evaluation Indicators):

[0185]

[0186] in, It is the average of all true values, y i This is the actual QoE value. is the model's predicted value, and L is the total number of samples. It measures the goodness of fit of the model to the data, and the closer the value is to 1, the better the fit.

[0187] ③ Point cloud block quality level allocation algorithm: This algorithm dynamically adjusts based on user perspective, occlusion level, and network conditions (such as bandwidth) to meet the target QoE (Quality of Experience) and optimize resource utilization. Main implementation:

[0188] • Based on information such as the user's field of view (FoV), network bandwidth, and frame rate requirements, the required quality score is predicted using a pre-trained MLP model.

[0189] • Use occlusion levels (0-3) to determine which point cloud blocks to transmit first and assign them the optimal quality level (QL0-QL3).

[0190] • Implement a "high-to-low" quality allocation strategy to minimize the amount of data transmitted while satisfying QoE requirements.

[0191]

[0192]

[0193] As can be seen from the above description, the embodiments of the present invention achieve the following technical effects:

[0194] Cross-modal feature extraction and fusion: Utilizing geometric features, color features, and projected image features, data fusion is performed through an attention mechanism to improve the accuracy of quality assessment. Multi-dimensional QoE modeling: Simultaneously considering network parameters (such as jitter, latency, and bandwidth) and content parameters (such as the geometric and color features of point clouds), a complex nonlinear relationship model is constructed to predict user subjective ratings. Dynamic resource allocation strategy: Based on the user's field of view, bandwidth limitations, and frame rate requirements, efficient point cloud video stitching and reconstruction are achieved, and transmission strategies are optimized to ensure user experience.

[0195] More accurate QoE prediction capability: By introducing deep learning models, more complex and detailed nonlinear mapping relationships can be captured, thereby improving the accuracy of QoE prediction. Comprehensive consideration of network factors: Not only focusing on the characteristics of the point cloud content itself, but also fully considering the impact of network state changes on user experience, achieving a more comprehensive service quality assessment. Personalized experience optimization: The transmission strategy can be dynamically adjusted according to different user behavior patterns and preferences, providing a more personalized viewing experience. Real-time response mechanism: With the help of pre-trained deep learning models, online computation requires almost no additional time overhead, ensuring low-latency response speed.

[0196] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A holographic video adaptive transmission method based on QoE guarantee, characterized in that, include: Real-time monitoring of network status, calculation of key network parameters such as jitter, latency, and available bandwidth to assess the current network environment; The quality score of point cloud video is calculated by combining geometric and color features using an attention mechanism. At the same time, the MANIQA model is introduced to extract depth features and noise and blur features are fused to improve the accuracy of quality assessment. By using deep learning methods, the network parameters and quality scores mentioned above are used as inputs to train a model that can predict users' subjective ratings, thus establishing a mapping relationship from objective indicators to subjective experience. Based on the user's field of view, bandwidth constraints, and frame rate requirements, and combined with the above-mentioned user subjective rating prediction results, the transmission strategy is dynamically adjusted. By controlling the field of view occlusion level and quantization level, efficient stitching and reconstruction of point cloud video is achieved, so as to optimize the user experience and improve the utilization of network resources, and realize high-quality and smooth point cloud video transmission.

2. The holographic video adaptive transmission method based on QoE assurance as described in claim 1, characterized in that, The network parameters are obtained by analyzing the time stamps of periodic probe packet transmission and reception, thereby calculating network latency and jitter. At the same time, traffic monitoring technology is used to estimate the current available bandwidth, providing basic data for subsequent transmission decisions. This enables the system to dynamically adjust the transmission strategy according to the actual network conditions, avoiding video stuttering or image quality degradation caused by insufficient bandwidth or network fluctuations.

3. The holographic video adaptive transmission method based on QoE assurance as described in claim 1, characterized in that, Through an attention mechanism, the system automatically identifies key areas that affect the visual experience and assigns them higher weights, thereby calculating quality scores more accurately, improving the precision of quality assessment, and providing reliable data support for subsequent user subjective rating predictions.

4. The holographic video adaptive transmission method based on QoE assurance as described in claim 1, characterized in that, Multilayer perceptrons or long short-term memory networks are employed to capture the complex nonlinear relationships between different features and to consider the changing trends over time. During training, a large amount of user viewing experience data under different network conditions is first collected, including objectively measured network parameters, quality scores, and user-reported MOS values. Subsequently, supervised learning methods are used to train the model, and cross-validation strategies are employed to optimize hyperparameters to ensure the model's generalization ability. After training, the model can predict possible subjective ratings from users based on real-time network conditions and video quality features, providing a basis for subsequent adaptive transmission strategies. This enables the system to dynamically adjust the transmission scheme under different network environments to maximize user experience.

5. The holographic video adaptive transmission method based on QoE assurance as described in claim 1, characterized in that, Based on the user's current field of view, the system prioritizes transmitting point cloud data within the visible area, while using lower quantization levels or direct occlusion for areas outside the field of view to reduce unnecessary data transmission. During the stitching and reconstruction of point cloud video, the system utilizes multiple quantization levels to differentiate the encoding of point cloud data in different regions. High-priority areas use lower quantization steps to maintain higher visual quality, while low-priority areas use larger quantization steps to reduce transmission bandwidth requirements. The system dynamically adjusts the selection of quantization levels based on real-time network bandwidth to ensure smooth video playback even with limited bandwidth. Finally, the quality score calculation module evaluates the quality of the reconstructed video, ensuring that the transmission strategy achieves efficient bandwidth utilization while guaranteeing user experience.

6. The holographic video adaptive transmission method based on QoE assurance as described in claim 1, characterized in that, Adaptive quality reconstruction divides the point cloud into blocks, assigning different quality levels to each block. First, an algorithm is used to determine the occlusion level of each block, identifying which blocks require high quality and which do not. Then, based on the occlusion level, the algorithm iterates through and allocates point cloud blocks with different quantization levels. The occlusion level represents the visibility of a point cloud block in the user's current field of view.

7. The holographic video adaptive transmission method based on QoE assurance as described in claim 6, characterized in that, The occlusion level is mainly based on the following factors: user's field of view direction, obtained through the head-mounted device or screen gaze point; angle between point cloud block and line of sight, calculated by the angle between the center of the point cloud block and the user's line of sight, the smaller the angle, the higher the priority; depth distance, the closer the point cloud block is to the user, the higher the priority; dynamic occlusion detection, using historical frame data or collision detection algorithms to determine whether it is occluded by other objects.

8. The holographic video adaptive transmission method based on QoE assurance as described in claim 1, characterized in that, Based on the occlusion level, point cloud blocks are mapped to the corresponding quantization level. The adaptive quality reconstruction includes: obtaining the center position of the point cloud block; calculating the angle between the viewing direction and the direction of the point cloud block; calculating the distance between the point cloud block and the user; determining whether it is occluded; comprehensively judging the occlusion level; mapping the occlusion level to the quantization level; and storing the results.

9. The holographic video adaptive transmission method based on QoE assurance as described in claim 1, characterized in that, The MLP model training and prediction steps include: data partitioning; data standardization; Bootstrap resampling; building the MLP model; compiling the model; model training; storing the model; model prediction; Bagging ensemble; calculating performance metrics; and returning the results.

10. The holographic video adaptive transmission method based on QoE assurance as described in claim 1, characterized in that, The point cloud block quality allocation steps include: initializing all blocks to the lowest quality; constructing occlusion level groups and sorting them from highest to lowest visibility; gradually reducing the occlusion level from the lowest level, iterating through the occlusion levels of each point cloud block; upgrading the quality level of the current block; backing up the current settings; calculating the total bandwidth consumption under the current configuration; if the required bit rate is greater than the available bandwidth; performing a rollback operation; if not, using an MLP model to predict the current QoE, determining if the predicted QoE is less than the target QoE, and continuing to attempt to upgrade if the target QoE is not reached, exiting early once the target is achieved; then, mapping IDs to quality level strings; and finally, evaluation.

Images