A reliable structure perception-based immersive video quality evaluation method and system

Abstract

The application relates to the technical field of computer vision, and discloses an immersive video quality evaluation method and system based on reliable structure perception, which respectively input a rendered immersive video, a residual patch sequence and a frame patch sequence into a feature extraction network and a 3D wavelet fusion module to respectively obtain a spatiotemporal quality feature of the rendered video, a spatiotemporal quality feature of the residual patch and a spatiotemporal quality feature of the frame patch, and an enhanced wavelet feature; the four features are fused through a gated residual to obtain a frequency domain enhanced spatiotemporal quality feature; a plurality of frames at the same time are sampled from a multi-view sequence, and a Page-4D model is used to obtain a viewpoint reliability score of each viewpoint; a K key viewpoint is selected through a greedy algorithm and input into a geometric relationship aggregation module to obtain a high-dimensional structure geometric relationship feature; finally, the frequency domain enhanced spatiotemporal quality feature and the high-dimensional structure geometric relationship feature are fused through a gated residual, and an immersive video overall quality score is obtained through MLP regression.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an immersive video quality assessment method and system based on reliable structure perception. Background Technology

[0002] With the development of virtual reality, augmented reality, free-viewpoint video, immersive communication, and 3D scene reconstruction technologies, immersive video has gradually become an important form of next-generation video media. Compared with traditional 2D video, immersive video has a larger field of view, a more complex multi-viewpoint structure, stronger interactivity, and richer 3D structural information. When watching immersive video, users not only focus on image clarity, texture integrity, and motion continuity from a single perspective, but are also affected by factors such as cross-viewpoint structural consistency, 3D geometric stability, rendering quality, and the experience of free-viewpoint switching. Immersive video is susceptible to various degradation factors during acquisition, encoding, compression, transmission, 3D reconstruction, and rendering. These distortions are not only reflected in spatial texture degradation but also in abnormal high-frequency information in the frequency domain, unstable temporal direction changes, and disruption of high-dimensional structural geometric relationships.

[0003] Existing no-reference video quality assessment methods typically use 2D video frames or short video clips as input, relying mainly on spatial texture features, local motion features, or semantic features from deep networks for quality prediction. While these methods can achieve some success in traditional video quality assessment tasks, they are insufficient in modeling multi-viewpoint geometric relationships, 3D structural inconsistencies, and rendering viewpoint distortions present in immersive videos. Furthermore, although some methods introduce frequency domain features, they often directly use high-frequency responses as quality degradation cues, failing to adequately distinguish between genuine structural high frequencies and pseudo-high-frequency interference caused by noise, artifacts, and unstable changes. This can easily lead to unreliable high-frequency information affecting the quality prediction results.

[0004] Therefore, the key issue to improve the accuracy and robustness of immersive video quality assessment is how to construct an immersive video quality assessment method that can simultaneously model the frequency domain enhanced spatiotemporal quality features and the high-dimensional structural geometric relationship features, and how to use residual fragment sequences and frame fragment sequences to reliably modulate wavelet high-frequency information. Summary of the Invention

[0005] The purpose of this invention is to solve the problems in the prior art.

[0006] The technical solution adopted by this invention to solve its technical problem is: to provide an immersive video quality assessment method based on reliable structure perception, comprising the following steps:

[0007] The immersive video data to be evaluated is obtained, including the rendered immersive video, the residual fragment sequence, the frame fragment sequence corresponding to the residual fragment, and the multi-frame multi-viewpoint sequence. The residual fragment sequence is obtained by cropping the first few regions with the largest local changes in the residual map between adjacent frames of the rendered immersive video, and the frame fragment sequence corresponding to the residual fragment is obtained by cropping the local regions in the rendered video frames that have the same spatial position as the residual fragment.

[0008] Features were extracted from the rendered immersive video, the residual fragment sequence, and the frame fragment sequence corresponding to the residual fragment to obtain the spatiotemporal quality features of the rendered video, the spatiotemporal quality features of the residual fragment, and the spatiotemporal quality features of the frame fragment.

[0009] The rendered immersive video is input into the 3D wavelet fusion module. High-frequency features are obtained through 3D discrete wavelet transform and divided into horizontal high-frequency sub-bands, vertical high-frequency sub-bands, diagonal high-frequency sub-bands and time high-frequency sub-bands. Then, the high-frequency sub-bands are weighted and fused to obtain wavelet features.

[0010] The residual fragment sequence and the corresponding frame fragment sequence are spliced ​​together at the same spatial and temporal location to obtain the spliced ​​fragment joint feature. The fragment joint feature is aligned with the wavelet feature in the spatiotemporal dimension and then subjected to 3D convolution to learn the local spatiotemporal variation law to generate a pseudo high-frequency suppression map. The pseudo high-frequency suppression map is then used to suppress and modulate the wavelet feature to obtain the enhanced wavelet feature.

[0011] The rendered video spatiotemporal quality features, residual fragment spatiotemporal quality features, frame fragment spatiotemporal quality features and enhanced wavelet features are input into the first gated residual fusion module to obtain frequency domain enhanced spatiotemporal quality features.

[0012] Several frames at the same time are sampled from a multi-frame, multi-viewpoint sequence and input into the Page-4D model for inference to obtain the viewpoint reliability score of each viewpoint at the same time frame. K key viewpoints are selected by a greedy algorithm. The Page-4D output of the K key viewpoints is input into the geometric relationship aggregation module to obtain high-dimensional structural geometric relationship features.

[0013] The frequency domain enhanced spatiotemporal quality features and high-dimensional structural geometric relationship features are input into the second gated residual fusion module to obtain the final fused features, and the overall quality score of the immersive video is obtained through MLP regression.

[0014] Preferably, the immersive video data is represented as follows:

[0015] ;

[0016] ;

[0017] ;

[0018] ;

[0019] in, This serves as an identifier for the rendered video and its frames. This represents the rendered immersive video sequence. Indicates the first Video frames after frame rendering Indicates the time frame index. Indicates the total number of video frames; Represents the residual fragment sequence. Indicates the first Residual fragments corresponding to the frame Indicates the first Video frames after frame rendering Indicates the first The selected frame A local area, Indicates the first The set of high-variance regions selected in the frame based on the residual intensity. This indicates that the selection is based on the residual strength of the local area. Operations in high-variable regions This indicates the number of residual fragments selected in each frame; This represents the frame fragment sequence corresponding to the residual fragment. Indicates the first Frame fragments corresponding to the frame Indicates from the first Frame fragments are obtained by cropping image blocks from the video frame after frame rendering, which are located in the same spatial position as the residual fragments. This represents a multi-frame, multi-view sequence. Indicates the first Frame number Images from multiple viewpoints Indicates the viewpoint index. This indicates the total number of viewpoints contained in each frame.

[0020] Preferably, the step of extracting features from the rendered immersive video, the residual fragment sequence, and the corresponding frame fragment sequence to obtain the rendered video spatiotemporal quality features, the residual fragment spatiotemporal quality features, and the frame fragment spatiotemporal quality features is performed using a Swin-T network and a fusion window module, as follows:

[0021] ;

[0022] ;

[0023] ;

[0024] in, This represents the spatiotemporal quality characteristics of the rendered video. This represents the Swin-T feature extraction function used for the post-rendered video branch. This represents the merge window module used for the rendered video branch. This represents the rendered immersive video sequence. This represents the spatiotemporal quality characteristics of residual fragments. This represents the Swin-T feature extraction function used for residual fragment sequence branching. This represents the fusion window module used for branching the residual fragment sequence; Represents a residual fragment sequence; Indicates the spatiotemporal quality characteristics of frame fragments. This represents the Swin-T feature extraction function used for frame fragment sequence branching. This represents the fusion window module used for frame fragment sequence branching. This represents the frame fragment sequence corresponding to the residual fragment.

[0025] Preferably, the rendered immersive video is input into a 3D wavelet fusion module, where high-frequency features are obtained through 3D discrete wavelet transform and divided into horizontal high-frequency sub-bands, vertical high-frequency sub-bands, diagonal high-frequency sub-bands, and temporal high-frequency sub-bands. Then, each high-frequency sub-band is weighted and fused to obtain wavelet features, represented as follows:

[0026] ;

[0027] ;

[0028] ;

[0029] in, This represents the set of high-frequency features obtained after 3D discrete wavelet transform. Represents the 3D discrete wavelet transform function. This represents the rendered immersive video sequence. Indicates the horizontal high-frequency sub-band. Indicates the vertical high-frequency sub-band. Indicates the diagonal high-frequency sub-band. Indicates the high-frequency subband of time, curly braces This represents a set consisting of multiple high-frequency subbands; where the high-frequency subband fusion weight vector... = This includes the weights of the corresponding horizontal, vertical, diagonal, and temporal high-frequency sub-bands. Represents the normalized exponential function, This represents the learnable weight matrix used to calculate the high-frequency subband fusion weights. This indicates a global average pooling operation. This represents the high-frequency characteristics formed by the combination of horizontal high-frequency subbands, vertical high-frequency subbands, diagonal high-frequency subbands, and time high-frequency subbands. This represents the learnable bias term used to calculate the high-frequency subband fusion weights; This represents the wavelet features obtained after weighted fusion. This represents the fusion weights corresponding to the horizontal high-frequency sub-bands. This represents the fusion weights corresponding to the vertical high-frequency sub-bands. This represents the fusion weights corresponding to the diagonal high-frequency subbands. This represents the fusion weight corresponding to the high-frequency subband in time.

[0030] Preferably, the residual fragment sequence and the corresponding frame fragment sequence are spliced ​​together at the same spatial and temporal location to obtain the spliced ​​fragment joint feature; the fragment joint feature is aligned with the wavelet feature in the spatiotemporal dimension, and the local spatiotemporal variation law is learned through 3D convolution to generate a pseudo-high-frequency suppression map, which is then used to suppress and modulate the wavelet feature to obtain the enhanced wavelet feature, represented as:

[0031] ;

[0032] ;

[0033] ;

[0034] ;

[0035] in, This represents the joint fragment feature obtained by splicing the residual fragment sequence and the frame fragment sequence at the same position. This indicates a splicing operation at the same spatial and temporal location. Represents the residual fragment sequence. This represents the frame fragment sequence corresponding to the residual fragment; This represents the joint features of fragments aligned with wavelet features in the spatiotemporal dimension. This indicates a spatiotemporal size alignment operation. Indicates wavelet features; This represents a pseudo-high-frequency suppression map. This represents the Sigmoid activation function. Indicates 3D convolution operation; Indicates enhanced wavelet features, This indicates an element-wise multiplication operation.

[0036] Preferably, the process of inputting the rendered video spatiotemporal quality features, residual fragment spatiotemporal quality features, frame fragment spatiotemporal quality features, and enhanced wavelet features into the first gated residual fusion module yields frequency domain enhanced spatiotemporal quality features, expressed as:

[0037] ;

[0038] ;

[0039] ;

[0040] in, This indicates the spatiotemporal quality fusion characteristics of the three paths. This represents the spatiotemporal quality characteristics of the rendered video. This represents the spatiotemporal quality characteristics of residual fragments. Indicates the spatiotemporal quality characteristics of frame fragments. This represents the fusion function of three spatiotemporal quality features; Indicates the first gating weight. This represents the Sigmoid activation function. This indicates the spatiotemporal quality fusion characteristics of the three paths. Indicates enhanced wavelet features, This represents the first gating weight matrix. Indicates the first gating bias term; This indicates the frequency domain enhancement of spatiotemporal quality characteristics. This represents the first residual scaling factor. This represents the enhanced wavelet feature mapping function. This indicates an element-wise multiplication operation.

[0041] Preferably, the step of sampling several frames at the same time from a multi-frame, multi-viewpoint sequence, inputting them into the Page-4D model for inference, and obtaining the viewpoint reliability score for each viewpoint at the same time frame is expressed as:

[0042] ;

[0043] ;

[0044] ;

[0045] in, Indicates the first Frame number Page-4D output from multiple viewpoints This represents the Page-4D inference function. Indicates the first Frame number Input image from each viewpoint, Represents a depth map. This represents a depth confidence plot. Represents a 3D point plot. Represents a 3D point confidence plot. This represents the reprojection error map. Indicates camera pose or camera parameters; Represents a reliability description vector. This indicates a global average pooling operation. This represents a geometric consistency metric function; Indicates the first Frame number Reliability score for each viewpoint This represents the Sigmoid activation function. This represents the transpose of the reliability score weight vector. This indicates the reliability score bias.

[0046] Preferably, K key viewpoints are selected using a greedy algorithm; the Page-4D outputs of the K key viewpoints are input into the geometric relationship aggregation module to obtain high-dimensional structural geometric relationship features, represented as follows:

[0047] ;

[0048] ;

[0049] ;

[0050] in, Indicates the first Frame number Geometric coding features of each viewpoint This indicates the Page-4D output encoding function. Indicates the first Frame number Page-4D output from multiple viewpoints; Indicates the first The frames were selected using a greedy algorithm. A set of key viewpoints This represents a greedy selection function for the key viewpoint. Indicates the first Frame number Reliability score for each viewpoint This indicates the total number of viewpoints contained in each frame. Indicates the number of key viewpoints selected; Represents the geometric relationship characteristics of high-dimensional structures. Aggregate functions representing geometric relationships This indicates the total number of video frames.

[0051] Preferably, the frequency domain enhanced spatiotemporal quality features and high-dimensional structural geometric relationship features are input into the second gated residual fusion module to obtain the final fused features, and the overall quality score of the immersive video is obtained through MLP regression, expressed as:

[0052] ;

[0053] ;

[0054] ;

[0055] in, Indicates the second gating weight. This represents the Sigmoid activation function. This represents the learnable weight matrix in the second gated residual fusion module. This represents the learnable bias term in the second gated residual fusion module. This indicates the frequency domain enhancement of spatiotemporal quality characteristics. Represents the geometric relationship characteristics of high-dimensional structures; Indicates the final fusion characteristics, This represents the second residual scaling factor. This represents element-wise multiplication. This represents a feature mapping function that maps high-dimensional structural geometric features to features of the same dimension as those used to enhance spatiotemporal quality in the frequency domain. This represents the predicted overall video quality score. This represents the regression function of the multilayer perceptron.

[0056] The present invention also provides an immersive video quality assessment system based on reliable structure awareness, comprising:

[0057] The data acquisition module acquires the immersive video data to be evaluated, including the rendered immersive video, the residual fragment sequence, the frame fragment sequence corresponding to the residual fragment, and the multi-frame multi-viewpoint sequence. The residual fragment sequence is obtained by cropping the first few regions with the largest local changes in the residual map between adjacent frames of the rendered immersive video, and the frame fragment sequence corresponding to the residual fragment is obtained by cropping the local regions in the rendered video frames that are at the same spatial position as the residual fragment.

[0058] The spatiotemporal quality feature extraction module extracts features from the rendered immersive video, the residual fragment sequence, and the frame fragment sequence corresponding to the residual fragment, respectively, to obtain the spatiotemporal quality features of the rendered video, the spatiotemporal quality features of the residual fragment, and the spatiotemporal quality features of the frame fragment.

[0059] The 3D wavelet fusion module inputs the rendered immersive video into the 3D wavelet fusion module, obtains high-frequency features through 3D discrete wavelet transform, and divides them into horizontal high-frequency sub-bands, vertical high-frequency sub-bands, diagonal high-frequency sub-bands, and time high-frequency sub-bands. Then, the high-frequency sub-bands are weighted and fused to obtain wavelet features.

[0060] The wavelet feature enhancement module splices the residual fragment sequence and the corresponding frame fragment sequence at the same spatial and temporal location to obtain the spliced ​​fragment joint feature. The fragment joint feature is aligned with the wavelet feature in the spatiotemporal dimension and learns the local spatiotemporal variation law through 3D convolution to generate a pseudo high-frequency suppression map. The pseudo high-frequency suppression map is then used to suppress and modulate the wavelet feature to obtain the enhanced wavelet feature.

[0061] The first gated residual fusion module inputs the rendered video spatiotemporal quality features, residual fragment spatiotemporal quality features, frame fragment spatiotemporal quality features and enhanced wavelet features into the first gated residual fusion module to obtain frequency domain enhanced spatiotemporal quality features;

[0062] The Page-4D inference module samples several frames at the same time from a multi-frame, multi-viewpoint sequence, inputs them into the Page-4D model for inference, obtains the viewpoint reliability score for each viewpoint at the same time frame, and selects K key viewpoints through a greedy algorithm; the Page-4D output of the K key viewpoints is input into the geometric relationship aggregation module to obtain high-dimensional structural geometric relationship features;

[0063] The final evaluation module inputs the frequency domain enhanced spatiotemporal quality features and high-dimensional structural geometric relationship features into the second gated residual fusion module to obtain the final fused features, and then obtains the overall quality score of the immersive video through MLP regression.

[0064] The present invention has the following beneficial effects:

[0065] (1) The present invention inputs the rendered immersive video, residual fragment sequence and frame fragment sequence into the Swing-T network and the fusion window module respectively for modeling, which can capture the overall rendered content, local change residual information and local content context information respectively, and improve the ability to represent the spatiotemporal quality degradation of immersive video.

[0066] (2) The present invention performs high-frequency modeling of the rendered immersive video in the horizontal, vertical, diagonal and temporal directions through the 3D wavelet fusion module, so that the model can simultaneously pay attention to the degradation of details in multiple spatial directions and the abnormal dynamic changes in the temporal direction, thereby improving the ability to perceive blur, noise, compression artifacts and temporal instability distortion.

[0067] (3) The present invention uses residual fragment sequence and frame fragment sequence to generate pseudo high frequency suppression map, so that the model can simultaneously refer to change information and content information at the same position, thereby determining whether the wavelet high frequency is a reliable structural high frequency or a pseudo high frequency caused by noise, artifacts or unstable changes;

[0068] (4) This invention utilizes Page-4D output to construct high-dimensional structural geometric relationship features. Through viewpoint reliability calculation and greedy key viewpoint selection, it can reduce the interference of low-quality viewpoints or redundant viewpoints on geometric modeling and improve the model's ability to perceive cross-viewpoint consistency, three-dimensional structural stability and reprojection reliability.

[0069] (5) This invention uses a two-level gated residual fusion mechanism to first fuse spatiotemporal quality features and enhanced wavelet features, and then fuse frequency domain enhanced spatiotemporal quality features and high-dimensional structural geometric relationship features, so that frequency domain information, spatiotemporal information and geometric information can adaptively complement each other, thereby improving the consistency between the immersive video quality prediction results and human subjective perception.

[0070] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. Attached Figure Description

[0071] Figure 1 This is a flowchart of an immersive video quality assessment method based on reliable structure perception, according to an embodiment of the present invention.

[0072] Figure 2 This is a schematic diagram illustrating the overall principle of an immersive video quality assessment method based on reliable structure perception, according to an embodiment of the present invention.

[0073] Figure 3 This is a schematic diagram of the 3D wavelet fusion module according to an embodiment of the present invention;

[0074] Figure 4 This is a schematic diagram of the structure of an immersive video quality assessment system based on reliable structure perception, according to an embodiment of the present invention. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0076] See Figure 1 As shown in the figure, an embodiment of the present invention provides an immersive video quality assessment method based on reliable structure perception, which includes the following steps:

[0077] S101, acquire the immersive video data to be evaluated, including the rendered immersive video, the residual fragment sequence, the frame fragment sequence corresponding to the residual fragment, and the multi-frame multi-viewpoint sequence; the residual fragment sequence is obtained by cropping the first few regions with the largest local changes in the residual map between adjacent frames of the rendered immersive video, and the frame fragment sequence corresponding to the residual fragment is obtained by cropping the local regions in the rendered video frames that have the same spatial position as the residual fragment;

[0078] S102, extract features from the rendered immersive video, the residual fragment sequence and the frame fragment sequence corresponding to the residual fragment, respectively, to obtain the spatiotemporal quality features of the rendered video, the spatiotemporal quality features of the residual fragment and the spatiotemporal quality features of the frame fragment.

[0079] S103, the rendered immersive video is input into the 3D wavelet fusion module, the high-frequency features are obtained through 3D discrete wavelet transform and divided into horizontal high-frequency sub-band, vertical high-frequency sub-band, diagonal high-frequency sub-band and time high-frequency sub-band, and then the high-frequency sub-bands are weighted and fused to obtain wavelet features;

[0080] S104, the residual fragment sequence and the frame fragment sequence corresponding to the residual fragment are spliced ​​together at the same spatial and temporal positions to obtain the spliced ​​fragment joint feature; the fragment joint feature is aligned with the wavelet feature in the spatiotemporal dimension, and the local spatiotemporal variation law is learned through 3D convolution to generate a pseudo high-frequency suppression map, and then the wavelet feature is suppressed and modulated using the pseudo high-frequency suppression map to obtain the enhanced wavelet feature.

[0081] S105, input the rendered video spatiotemporal quality features, residual fragment spatiotemporal quality features, frame fragment spatiotemporal quality features and enhanced wavelet features into the first gated residual fusion module to obtain frequency domain enhanced spatiotemporal quality features;

[0082] S106: Sample several frames at the same time from a multi-frame, multi-viewpoint sequence, input them into the Page-4D model for inference, obtain the viewpoint reliability score for each viewpoint at the same time frame, and select K key viewpoints through a greedy algorithm; input the Page-4D output of the K key viewpoints into the geometric relationship aggregation module to obtain high-dimensional structural geometric relationship features;

[0083] S107 inputs the frequency domain enhanced spatiotemporal quality features and high-dimensional structural geometric relationship features into the second gated residual fusion module to obtain the final fusion features, and obtains the overall quality score of the immersive video through MLP regression.

[0084] Specifically, in this embodiment, the residual fragment sequence described in S101 is obtained by cropping the first few regions with the largest local changes in the residual map between adjacent frames of the rendered immersive video. Specifically, it is obtained by selecting the first few local regions and splicing them together after sorting them from high to low local residual intensity. The frame fragment sequence corresponding to the residual fragment is obtained by cropping the local regions in the rendered video frames that have the same spatial position as the residual fragment. The input data is represented as follows:

[0085] ;

[0086] ;

[0087] ;

[0088] ;

[0089] in, This represents the rendered immersive video sequence. Indicates the first Video frames after frame rendering; Indicates the time frame index; Indicates the total number of video frames; Represents a residual fragment sequence; Indicates the first Residual fragments corresponding to the frame; This represents the frame fragment sequence corresponding to the residual fragment; Indicates the first Frame fragments corresponding to the frame; This represents the first frame of the rendered video. Frame and the Residual map between frames; Indicates the first The selected frame A local area; Indicates the first The set of high-variation regions selected in the frame based on the residual intensity; This indicates that the selection is based on the residual strength of the local area. Operations in high-variability regions; This indicates the number of residual fragments selected in each frame; Indicates from the first Frame fragments are obtained by cropping image blocks from the video frame after frame rendering, which are located in the same spatial position as the residual fragments. This represents a multi-frame, multi-view sequence; Indicates the first Frame number Images from multiple viewpoints; Indicates the viewpoint index; This indicates the total number of viewpoints contained in each frame.

[0090] Specifically, in this embodiment, the spatiotemporal quality features of the rendered video, the spatiotemporal quality features of the residual fragments, and the spatiotemporal quality features of the frame fragments described in S102 are represented as follows:

[0091] ;

[0092] ;

[0093] ;

[0094] in, This represents the spatiotemporal quality characteristics of the rendered video. This represents the spatiotemporal quality characteristics of residual fragments; Indicates the spatiotemporal quality characteristics of frame fragments; This represents the Swin-T feature extraction function used for the post-rendered video branch; This represents the Swin-T feature extraction function used for branching residual fragment sequences; This represents the Swin-T feature extraction function used for frame fragment sequence branching; the above three Swin-T feature extraction functions extract the corresponding spatiotemporal features through Swin-T hierarchical spatiotemporal window modeling; This represents the blending window module used for the rendered video branch; This represents the fusion window module used for branching the residual fragment sequence; This represents the fusion window module used for the frame fragment sequence branch; the above three fusion window modules adopt the same feature fusion method to associate and fuse the window features extracted from the corresponding Swin-T branch to obtain the corresponding spatiotemporal quality features; This represents the rendered immersive video sequence. Represents a residual fragment sequence; This represents the frame fragment sequence corresponding to the residual fragment.

[0095] Specifically, in this embodiment, the process of obtaining wavelet features in S103 is as follows:

[0096] ;

[0097] ;

[0098] ;

[0099] in, This represents the set of high-frequency features obtained after 3D discrete wavelet transform. Represents the 3D discrete wavelet transform function; This represents the rendered immersive video sequence. Indicates the horizontal high-frequency sub-band; Indicates the vertical high-frequency sub-band; Indicates the diagonal high-frequency sub-band; Indicates a high-frequency subband in time; curly braces It represents a set consisting of multiple high-frequency subbands; This represents the high-frequency subband fusion weight vector; This represents the normalized exponential function; This represents the learnable weight matrix used to calculate the high-frequency subband fusion weights; This indicates a global average pooling operation; It represents the high-frequency characteristics formed by the combination of horizontal high-frequency sub-bands, vertical high-frequency sub-bands, diagonal high-frequency sub-bands, and time high-frequency sub-bands; This represents the learnable bias term used to calculate the high-frequency subband fusion weights; This represents the wavelet features obtained after weighted fusion; This represents the fusion weights corresponding to the horizontal high-frequency sub-bands; This represents the fusion weights corresponding to the vertical high-frequency sub-bands; This represents the fusion weights corresponding to the diagonal high-frequency subbands; This represents the fusion weight corresponding to the high-frequency subband in time.

[0100] Specifically, in this embodiment, S104 generates a pseudo-high-frequency suppression map, and uses the pseudo-high-frequency suppression map to suppress and modulate the wavelet features, resulting in the enhanced wavelet features. The process is as follows:

[0101] ;

[0102] ;

[0103] ;

[0104] ;

[0105] in, This represents the joint fragment feature obtained by splicing the residual fragment sequence and the frame fragment sequence at the same position; This indicates a splicing operation at the same spatial and temporal location; Represents a residual fragment sequence; This represents the frame fragment sequence corresponding to the residual fragment; This represents the joint fragment features aligned with wavelet features in the spatiotemporal dimension; This indicates a spatiotemporal size alignment operation; This represents the wavelet features obtained after weighted fusion; This represents a pseudo-high-frequency suppression map. The closer the value is to 0, the more reliable the high-frequency information at the current location. This represents the Sigmoid activation function, used to constrain the values ​​of the pseudo-high-frequency suppression map to between 0 and 1; Indicates 3D convolution operation; Indicates enhanced wavelet features; This indicates an element-wise multiplication operation.

[0106] Specifically, in this embodiment, the first gated residual fusion module in S105 performs the following steps: concatenates the three spatiotemporal quality fusion features with the enhanced wavelet features, obtains the first gate weight through linear mapping and Sigmoid activation, and then uses the gate weight to weight the enhanced wavelet features before fusing them with the three spatiotemporal quality fusion features in the form of residuals.

[0107] Specifically, the process of gated residual fusion of the three spatiotemporal quality features and the enhanced wavelet features to obtain the frequency domain enhanced spatiotemporal quality features is as follows:

[0108] ;

[0109] ;

[0110] ;

[0111] in, This indicates the spatiotemporal quality fusion characteristics of the three paths; This represents the spatiotemporal quality characteristics of the rendered video. This represents the spatiotemporal quality characteristics of residual fragments; Indicates the spatiotemporal quality characteristics of frame fragments; This represents the fusion function of three spatiotemporal quality features, which obtains the fused features of three spatiotemporal quality features through feature concatenation and mapping fusion. Indicates the first gating weight; This represents the first gating weight matrix; Indicates the first gating bias term; This represents the Sigmoid activation function; Indicates enhanced wavelet features; This indicates frequency domain-enhanced spatiotemporal quality characteristics; This represents the first residual scaling factor; This represents the enhanced wavelet feature mapping function, which is used to perform size alignment and feature dimension mapping on the enhanced wavelet features so that they have the same feature dimension as the three spatiotemporal quality fusion features; This indicates an element-wise multiplication operation.

[0112] Specifically, in this embodiment, S106 samples several frames at the same time from the multi-viewpoint sequence, inputs them into the Page-4D model for inference, and obtains the viewpoint reliability score for each viewpoint at the same time frame. The calculation formula is as follows:

[0113] ;

[0114] ;

[0115] ;

[0116] in, Indicates the first Frame number Page-4D output from multiple viewpoints; Represents the Page-4D inference function; Indicates the first Frame number Input image from one viewpoint; Represents a depth map; Represents a depth confidence plot; Represents a 3D point plot; Represents a 3D point confidence plot; This represents the reprojection error diagram; Indicates camera pose or camera parameters; Represents a reliability description vector; This indicates a global average pooling operation; This represents the Sigmoid activation function; Indicates the first Frame number Reliability score for each viewpoint; This represents the geometric consistency metric function, which calculates cross-viewpoint reprojection consistency based on 3D points and corresponding camera parameters, and uses the reprojection error to characterize the degree of geometric consistency between different viewpoints. This represents the transpose of the reliability score weight vector; This indicates the reliability score bias.

[0117] Specifically, in this embodiment, S106 selects K key viewpoints using a greedy algorithm and aggregates their geometric relationships to obtain high-dimensional structural geometric relationship features, as follows:

[0118] ;

[0119] ;

[0120] ;

[0121] in, Indicates the first Frame number Geometric coding features of each viewpoint; This represents the Page-4D output encoding function, which encodes the depth, depth confidence, 3D points, 3D point confidence, reprojection error, and camera parameters of the Page-4D output to obtain the geometric encoding features of the corresponding viewpoint. Indicates the first Frame number Page-4D output from multiple viewpoints; Indicates the first The frames were selected using a greedy algorithm. A set of key viewpoints; This represents a greedy selection function for key viewpoints; Indicates the first Frame number Reliability score for each viewpoint; This indicates the total number of viewpoints contained in each frame; Indicates the number of key viewpoints selected; The geometric relationship aggregation function is used to aggregate cross-viewpoint relationships of the geometric encoding features of the selected key viewpoints, and further aggregate the geometric information at multiple sampling times to obtain high-dimensional structural geometric relationship features. Represents the geometric relationship characteristics of high-dimensional structures; This indicates the total number of video frames.

[0122] Specifically, the second gated residual fusion module described in S107 of this embodiment is as follows: the frequency domain enhanced spatiotemporal quality features and high-dimensional structural geometric relationship features are spliced ​​together, and a second gate weight is obtained through linear mapping and Sigmoid activation. The high-dimensional structural geometric relationship features are then weighted using the gate weight and fused in the form of residuals.

[0123] Specifically, the formulas for calculating the final fused features and the overall video quality score are as follows:

[0124] ;

[0125] ;

[0126] ;

[0127] in, Indicates the second gating weight; This represents the Sigmoid activation function; This represents the learnable weight matrix in the second gated residual fusion module; This represents the learnable bias term in the second gated residual fusion module; Indicates the final fusion characteristics; This indicates frequency domain-enhanced spatiotemporal quality characteristics; Represents the geometric relationship characteristics of high-dimensional structures; This represents the second residual scaling factor; This represents an element-wise multiplication operation; This represents a feature mapping function that maps high-dimensional structural geometric features to features of the same dimension as those used to enhance spatiotemporal quality in the frequency domain. This represents the overall quality score of the predicted video. This represents the regression function of the multilayer perceptron.

[0128] Specifically, Figure 2 This invention demonstrates the overall architecture of the immersive video quality assessment model based on reliable structure awareness proposed in this paper, covering the complete process from multi-source input acquisition, three-way spatiotemporal quality feature extraction, 3D wavelet high-frequency modeling, pseudo-high-frequency suppression modulation, high-dimensional structural geometric relationship modeling to quality regression. The process involves inputting the rendered immersive video, residual fragment sequence, and frame fragment sequence into a Swin-T network and a fusion window module, respectively, to extract the spatiotemporal quality features of the rendered video, residual fragments, and frame fragments. The rendered immersive video, residual fragment sequence, and frame fragment sequence are then input into a 3D wavelet fusion module to obtain enhanced wavelet features. Gated residual fusion is performed on the rendered video spatiotemporal quality features, residual fragment spatiotemporal quality features, frame fragment spatiotemporal quality features, and enhanced wavelet features to obtain frequency domain enhanced spatiotemporal quality features. Simultaneously, several frames at the same time are sampled from the multi-viewpoint sequence and input into a Page-4D model for inference to obtain the viewpoint reliability score for each viewpoint at the same time. A greedy algorithm is used to select K key viewpoints, and the Page-4D outputs of these K key viewpoints are input into a geometric relationship aggregation module to obtain high-dimensional structural geometric relationship features. Finally, gated residual fusion is performed on the frequency domain enhanced spatiotemporal quality features and the high-dimensional structural geometric relationship features, and MLP regression is used to obtain the overall quality score of the immersive video. Figure 3 Detailed description Figure 2 The 3D wavelet fusion module in this invention focuses on multi-directional high-frequency modeling and pseudo-high-frequency suppression modulation mechanisms. This module first performs a 3D discrete wavelet transform on the rendered immersive video to obtain horizontal, vertical, diagonal, and temporal high-frequency subbands. Wavelet features are then obtained through weighted fusion. To distinguish between real structural high frequencies and pseudo-high frequencies caused by noise, artifacts, or unstable changes, this invention splices the residual fragment sequence and the corresponding frame fragment sequence at the same spatial and temporal location. The spliced ​​fragment joint features are then aligned with the wavelet features in the spatiotemporal dimension. After learning local spatiotemporal variation patterns through 3D convolution, a pseudo-high-frequency suppression map is generated. This map is then used to suppress and modulate the wavelet features, resulting in enhanced wavelet features.

[0129] Specifically, the main objective of this invention is to provide an immersive video quality assessment method based on reliable structure perception. This method organically integrates frequency-domain enhanced spatiotemporal quality features with high-dimensional structural geometric relationship features to construct a referenceless immersive video quality prediction model. This allows for automated and high-precision evaluation of the overall perceived quality of immersive videos without the need for a reference video. Compared to existing technologies, this invention not only focuses on texture degradation, motion changes, and high-frequency anomalies in the rendered video, but also further incorporates high-dimensional structural geometric information contained in the Page-4D output. Reliable viewpoint selection and geometric relationship aggregation enhance the model's ability to perceive 3D structural distortions in immersive videos. Simultaneously, this invention utilizes a pseudo-high-frequency suppression map to reliably modulate wavelet high-frequency features, reducing the interference of noise, artifacts, and unstable changes on quality prediction and improving the consistency between the referenceless immersive video quality assessment results and human subjective perception.

[0130] Specifically, in this embodiment, the proposed immersive video quality assessment model based on reliable structure awareness can be built using the PyTorch deep learning framework and tested using an NVIDIA RTX A6000 GPU. During training, the rendered immersive video, residual fragment sequences, frame fragment sequences, and multi-frame multi-view sequences are first uniformly read and preprocessed to maintain correspondence between different input branches in the temporal, spatial, and sample index dimensions. The training epochs are 100, the initial learning rate is set to 0.00001, and the Adam optimizer is used for training. The experimental datasets are public datasets such as LIVE-360, CVIQ, and VQA-ODV. 60% of the datasets are used for training, 20% for validation, and the remaining 20% ​​for testing. Spearman Rank Correlation Coefficient (SROCC), Pearson Linear Correlation Coefficient (PLCC), Kendall Rank Correlation Coefficient (KROCC), and Root Mean Square Error (RMSE) are selected to evaluate the performance of the model. Higher values ​​for SROCC, PLCC, and KROCC indicate better consistency between the prediction results and the subjective quality scores, while lower values ​​for RMSE indicate smaller prediction errors.

[0131] Comparative experiments were conducted on embodiments of the present invention, and the experimental results are shown in the table below:

[0132] As can be seen from the table data, the method proposed in this invention achieves the best performance in all four evaluation indicators by jointly modeling the frequency domain enhancement spatiotemporal quality features and high-dimensional structural geometric relationship features, verifying the effectiveness of synergistically characterizing the quality of immersive video from two dimensions: spatiotemporal degradation representation and multi-viewpoint geometric relationship.

[0133] like Figure 4As shown, an embodiment of the present invention provides an immersive video quality assessment system based on reliable structure awareness, comprising:

[0134] The data acquisition module 401 acquires the immersive video data to be evaluated, including the rendered immersive video, the residual fragment sequence, the frame fragment sequence corresponding to the residual fragment, and the multi-frame multi-viewpoint sequence; the residual fragment sequence is obtained by cropping the first few regions with the largest local changes in the residual map between adjacent frames of the rendered immersive video, and the frame fragment sequence corresponding to the residual fragment is obtained by cropping the local regions in the rendered video frames that have the same spatial position as the residual fragment.

[0135] The spatiotemporal quality feature extraction module 402 extracts features from the rendered immersive video, the residual fragment sequence, and the frame fragment sequence corresponding to the residual fragment, respectively, to obtain the spatiotemporal quality features of the rendered video, the spatiotemporal quality features of the residual fragment, and the spatiotemporal quality features of the frame fragment.

[0136] The 3D wavelet fusion module 403 inputs the rendered immersive video into the 3D wavelet fusion module, obtains high-frequency features through 3D discrete wavelet transform and divides them into horizontal high-frequency sub-bands, vertical high-frequency sub-bands, diagonal high-frequency sub-bands and time high-frequency sub-bands, and then performs weighted fusion of each high-frequency sub-band to obtain wavelet features;

[0137] The wavelet feature enhancement module 404 splices the residual fragment sequence and the corresponding frame fragment sequence at the same spatial and temporal positions to obtain the spliced ​​fragment joint feature. The fragment joint feature is aligned with the wavelet feature in the spatiotemporal dimension and learns the local spatiotemporal variation law through 3D convolution to generate a pseudo high-frequency suppression map. The pseudo high-frequency suppression map is then used to suppress and modulate the wavelet feature to obtain the enhanced wavelet feature.

[0138] The first gated residual fusion module 405 inputs the rendered video spatiotemporal quality features, residual fragment spatiotemporal quality features, frame fragment spatiotemporal quality features and enhanced wavelet features into the first gated residual fusion module to obtain frequency domain enhanced spatiotemporal quality features;

[0139] Page-4D inference module 406 samples several frames at the same time from a multi-frame, multi-viewpoint sequence, inputs them into the Page-4D model for inference, obtains the viewpoint reliability score for each viewpoint at the same time frame, and selects K key viewpoints through a greedy algorithm; the Page-4D output of the K key viewpoints is input into the geometric relationship aggregation module to obtain high-dimensional structural geometric relationship features;

[0140] The final evaluation module 407 inputs the frequency domain enhanced spatiotemporal quality features and high-dimensional structural geometric relationship features into the second gated residual fusion module to obtain the final fusion features, and obtains the overall quality score of the immersive video through MLP regression.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating the quality of immersive video based on reliable structure perception, characterized in that, Includes the following steps: The immersive video data to be evaluated is obtained, including the rendered immersive video, the residual fragment sequence, the frame fragment sequence corresponding to the residual fragment, and the multi-frame multi-viewpoint sequence. The residual fragment sequence is obtained by cropping the first few regions with the largest local changes in the residual map between adjacent frames of the rendered immersive video, and the frame fragment sequence corresponding to the residual fragment is obtained by cropping the local regions in the rendered video frames that have the same spatial position as the residual fragment. Features were extracted from the rendered immersive video, the residual fragment sequence, and the frame fragment sequence corresponding to the residual fragment to obtain the spatiotemporal quality features of the rendered video, the spatiotemporal quality features of the residual fragment, and the spatiotemporal quality features of the frame fragment. The rendered immersive video is input into the 3D wavelet fusion module. High-frequency features are obtained through 3D discrete wavelet transform and divided into horizontal high-frequency sub-bands, vertical high-frequency sub-bands, diagonal high-frequency sub-bands and time high-frequency sub-bands. Then, the high-frequency sub-bands are weighted and fused to obtain wavelet features. The residual fragment sequence and the frame fragment sequence corresponding to the residual fragment are spliced ​​together at the same spatial and temporal locations to obtain the spliced ​​fragment joint features; The fragment joint features are aligned with wavelet features in the spatiotemporal dimension, and the local spatiotemporal variation rules are learned through 3D convolution to generate a pseudo high-frequency suppression map. The pseudo high-frequency suppression map is then used to suppress and modulate the wavelet features to obtain enhanced wavelet features. The rendered video spatiotemporal quality features, residual fragment spatiotemporal quality features, frame fragment spatiotemporal quality features and enhanced wavelet features are input into the first gated residual fusion module to obtain frequency domain enhanced spatiotemporal quality features. Several frames at the same time are sampled from a multi-frame, multi-viewpoint sequence and input into the Page-4D model for inference to obtain the viewpoint reliability score of each viewpoint at the same time frame. K key viewpoints are selected by a greedy algorithm. The Page-4D output of the K key viewpoints is input into the geometric relationship aggregation module to obtain high-dimensional structural geometric relationship features. The frequency domain enhanced spatiotemporal quality features and high-dimensional structural geometric relationship features are input into the second gated residual fusion module to obtain the final fused features, and the overall quality score of the immersive video is obtained through MLP regression.

2. The immersive video quality assessment method based on reliable structure perception according to claim 1, characterized in that, The immersive video data is represented as follows: ； ； ； ； in, This serves as an identifier for the rendered video and its frames. This represents the rendered immersive video sequence. Indicates the first Video frames after frame rendering Indicates the time frame index. Indicates the total number of video frames; Represents the residual fragment sequence. Indicates the first Residual fragments corresponding to the frame Indicates the first Video frames after frame rendering Indicates the first The selected frame A local area, Indicates the first The set of high-variance regions selected in the frame based on the residual intensity. This indicates that the selection is based on the residual strength of the local area. Operations in high-variability regions This indicates the number of residual fragments selected in each frame; This represents the frame fragment sequence corresponding to the residual fragment. Indicates the first Frame fragments corresponding to the frame Indicates from the first Frame fragments are obtained by cropping image blocks from the video frame after frame rendering, which are located in the same spatial position as the residual fragments. This represents a multi-frame, multi-view sequence. Indicates the first Frame number Images from multiple viewpoints Indicates the viewpoint index. This indicates the total number of viewpoints contained in each frame.

3. The immersive video quality assessment method based on reliable structure perception according to claim 1, characterized in that, The process involves extracting features from the rendered immersive video, the residual fragment sequence, and the corresponding frame fragment sequence to obtain the spatiotemporal quality features of the rendered video, the residual fragments, and the frame fragments. These features are represented using a Swin-T network and a fusion window module. ； ； ； in, This represents the spatiotemporal quality characteristics of the rendered video. This represents the Swin-T feature extraction function used for the post-rendered video branch. This represents the merge window module used for the rendered video branch. This represents the rendered immersive video sequence. This represents the spatiotemporal quality characteristics of residual fragments. This represents the Swin-T feature extraction function used for residual fragment sequence branching. This represents the fusion window module used for branching the residual fragment sequence; Represents a residual fragment sequence; Indicates the spatiotemporal quality characteristics of frame fragments. This represents the Swin-T feature extraction function used for frame fragment sequence branching. This represents the fusion window module used for frame fragment sequence branching. This represents the frame fragment sequence corresponding to the residual fragment.

4. The immersive video quality assessment method based on reliable structure perception according to claim 1, characterized in that, The rendered immersive video is input into a 3D wavelet fusion module. High-frequency features are obtained through 3D discrete wavelet transform and divided into horizontal, vertical, diagonal, and temporal high-frequency sub-bands. Then, weighted fusion of each high-frequency sub-band yields wavelet features, represented as follows: ； ； ； in, This represents the set of high-frequency features obtained after 3D discrete wavelet transform. Represents the 3D discrete wavelet transform function. This represents the rendered immersive video sequence. Indicates the horizontal high-frequency sub-band. Indicates the vertical high-frequency sub-band. Indicates the diagonal high-frequency sub-band. Indicates the high-frequency subband of time, curly braces This represents a set consisting of multiple high-frequency subbands; where the high-frequency subband fusion weight vector... = This includes the weights of the corresponding horizontal, vertical, diagonal, and temporal high-frequency sub-bands. Represents the normalized exponential function, This represents the learnable weight matrix used to calculate the high-frequency subband fusion weights. This indicates a global average pooling operation. This represents the high-frequency characteristics formed by the combination of horizontal high-frequency subbands, vertical high-frequency subbands, diagonal high-frequency subbands, and time high-frequency subbands. This represents the learnable bias term used to calculate the high-frequency subband fusion weights; This represents the wavelet features obtained after weighted fusion. This represents the fusion weights corresponding to the horizontal high-frequency sub-bands. This represents the fusion weights corresponding to the vertical high-frequency sub-bands. This represents the fusion weights corresponding to the diagonal high-frequency subbands. This represents the fusion weight corresponding to the high-frequency subband in time.

5. The immersive video quality assessment method based on reliable structure perception according to claim 1, characterized in that, The residual fragment sequence and the frame fragment sequence corresponding to the residual fragment are spliced ​​together at the same spatial and temporal positions to obtain the spliced ​​fragment joint features; The fragment joint features are aligned with wavelet features in the spatiotemporal dimension, and then subjected to 3D convolution to learn the local spatiotemporal variation patterns, generating a pseudo-high-frequency suppression map. This pseudo-high-frequency suppression map is then used to suppress and modulate the wavelet features, resulting in enhanced wavelet features, represented as follows: ； ； ； ； in, This represents the joint fragment feature obtained by splicing the residual fragment sequence and the frame fragment sequence at the same position. This indicates a splicing operation at the same spatial and temporal location. Represents the residual fragment sequence. This represents the frame fragment sequence corresponding to the residual fragment; This represents the joint features of fragments aligned with wavelet features in the spatiotemporal dimension. This indicates a spatiotemporal size alignment operation. Indicates wavelet features; This represents a pseudo-high-frequency suppression map. This represents the Sigmoid activation function. Indicates 3D convolution operation; Indicates enhanced wavelet features, This indicates an element-wise multiplication operation.

6. The immersive video quality assessment method based on reliable structure perception according to claim 1, characterized in that, The rendered video spatiotemporal quality features, residual fragment spatiotemporal quality features, frame fragment spatiotemporal quality features, and enhanced wavelet features are input into the first gated residual fusion module to obtain the frequency domain enhanced spatiotemporal quality features, which are expressed as follows: ； ； ； in, This indicates the spatiotemporal quality fusion characteristics of the three paths. This represents the spatiotemporal quality characteristics of the rendered video. This represents the spatiotemporal quality characteristics of residual fragments. Indicates the spatiotemporal quality characteristics of frame fragments. This represents the fusion function of three spatiotemporal quality features; Indicates the first gating weight. This represents the Sigmoid activation function. This indicates the spatiotemporal quality fusion characteristics of the three paths. Indicates enhanced wavelet features, This represents the first gating weight matrix. Indicates the first gating bias term; This indicates the frequency domain enhancement of spatiotemporal quality characteristics. This represents the first residual scaling factor. This represents the enhanced wavelet feature mapping function. This indicates an element-wise multiplication operation.

7. The immersive video quality assessment method based on reliable structure perception according to claim 1, characterized in that, The process involves sampling several frames at the same time from a multi-frame, multi-viewpoint sequence, inputting them into the Page-4D model for inference, and obtaining the viewpoint reliability score for each viewpoint at the same time frame, expressed as: ； ； ； in, Indicates the first Frame number Page-4D output from multiple viewpoints This represents the Page-4D inference function. Indicates the first Frame number Input image from each viewpoint, Represents a depth map. This represents a depth confidence plot. Represents a 3D point plot. Represents a 3D point confidence plot. This represents the reprojection error map. Indicates camera pose or camera parameters; Represents a reliability description vector. This indicates a global average pooling operation. This represents a geometric consistency metric function; Indicates the first Frame number Reliability score for each viewpoint This represents the Sigmoid activation function. This represents the transpose of the reliability score weight vector. This indicates the reliability score bias.

8. The immersive video quality assessment method based on reliable structure perception according to claim 1, characterized in that, The process involves selecting K key viewpoints using a greedy algorithm; then inputting the Page-4D output of these K key viewpoints into a geometric relationship aggregation module to obtain high-dimensional structural geometric relationship features, represented as follows: ； ； ； in, Indicates the first Frame number Geometric coding features of each viewpoint This indicates the Page-4D output encoding function. Indicates the first Frame number Page-4D output from multiple viewpoints; Indicates the first The frames were selected using a greedy algorithm. A set of key viewpoints This represents a greedy selection function for the key viewpoint. Indicates the first Frame number Reliability score for each viewpoint This indicates the total number of viewpoints contained in each frame. Indicates the number of key viewpoints selected; Represents the geometric relationship characteristics of high-dimensional structures. Aggregate functions representing geometric relationships This indicates the total number of video frames.

9. The immersive video quality assessment method based on reliable structure perception according to claim 1, characterized in that, The frequency domain enhanced spatiotemporal quality features and high-dimensional structural geometric relationship features are input into the second gated residual fusion module to obtain the final fused features. The overall quality score of the immersive video is then obtained through MLP regression, and is expressed as follows: ； ； ； in, Indicates the second gating weight. This represents the Sigmoid activation function. This represents the learnable weight matrix in the second gated residual fusion module. This represents the learnable bias term in the second gated residual fusion module. This indicates the frequency domain enhancement of spatiotemporal quality characteristics. Represents the geometric relationship characteristics of high-dimensional structures; Indicates the final fusion characteristics, This represents the second residual scaling factor. This represents element-wise multiplication. This represents a feature mapping function that maps high-dimensional structural geometric features to features of the same dimension as those used to enhance spatiotemporal quality in the frequency domain. This represents the predicted overall video quality score. This represents the regression function of the multilayer perceptron.

10. An immersive video quality assessment system based on reliable structure perception, characterized in that, include: The data acquisition module acquires the immersive video data to be evaluated, including the rendered immersive video, the residual fragment sequence, the frame fragment sequence corresponding to the residual fragment, and the multi-frame multi-viewpoint sequence. The residual fragment sequence is obtained by cropping the first few regions with the largest local changes in the residual map between adjacent frames of the rendered immersive video, and the frame fragment sequence corresponding to the residual fragment is obtained by cropping the local regions in the rendered video frames that are at the same spatial position as the residual fragment. The spatiotemporal quality feature extraction module extracts features from the rendered immersive video, the residual fragment sequence, and the frame fragment sequence corresponding to the residual fragment, respectively, to obtain the spatiotemporal quality features of the rendered video, the spatiotemporal quality features of the residual fragment, and the spatiotemporal quality features of the frame fragment. The 3D wavelet fusion module inputs the rendered immersive video into the 3D wavelet fusion module, obtains high-frequency features through 3D discrete wavelet transform, and divides them into horizontal high-frequency sub-bands, vertical high-frequency sub-bands, diagonal high-frequency sub-bands, and time high-frequency sub-bands. Then, the high-frequency sub-bands are weighted and fused to obtain wavelet features. The wavelet feature enhancement module splices the residual fragment sequence and the frame fragment sequence corresponding to the residual fragment at the same spatial and temporal positions to obtain the spliced ​​fragment joint features. The fragment joint features are aligned with wavelet features in the spatiotemporal dimension, and the local spatiotemporal variation rules are learned through 3D convolution to generate a pseudo high-frequency suppression map. The pseudo high-frequency suppression map is then used to suppress and modulate the wavelet features to obtain enhanced wavelet features. The first gated residual fusion module inputs the rendered video spatiotemporal quality features, residual fragment spatiotemporal quality features, frame fragment spatiotemporal quality features and enhanced wavelet features into the first gated residual fusion module to obtain frequency domain enhanced spatiotemporal quality features; The Page-4D inference module samples several frames at the same time from a multi-frame, multi-viewpoint sequence, inputs them into the Page-4D model for inference, obtains the viewpoint reliability score for each viewpoint at the same time frame, and selects K key viewpoints through a greedy algorithm; the Page-4D output of the K key viewpoints is input into the geometric relationship aggregation module to obtain high-dimensional structural geometric relationship features; The final evaluation module inputs the frequency domain enhanced spatiotemporal quality features and high-dimensional structural geometric relationship features into the second gated residual fusion module to obtain the final fused features, and then obtains the overall quality score of the immersive video through MLP regression.

Images