HDR video quality evaluation method and device based on residual time correction, equipment and medium

Abstract

The application discloses an HDR video quality evaluation method and device based on residual timing correction, equipment and medium, relates to the field of video processing, and comprises the following steps: constructing a first training set and a target view corresponding to each video frame based on training video data, using an initial evaluator to obtain a first frame-by-frame basic quality response and updating the target evaluator; using an initial video quality evaluation model loaded with the target evaluator to obtain a second frame-by-frame basic quality response, a frame-by-frame spatial hidden feature and a frame-by-frame space-time hidden feature of each HDR continuous frame segment; determining a corresponding frame-by-frame residual correction amount by using a residual prediction head of the initial video quality evaluation model and obtaining a training quality score; obtaining a target video quality evaluation model based on the training quality score and a second subjective quality score; and obtaining a target video quality evaluation result corresponding to the HDR video to be evaluated by using the target video quality evaluation model. The application improves the accuracy and stability of HDR video quality evaluation.

Description

Technical Field

[0001] This invention relates to the field of video processing, and in particular to a method, apparatus, device, and medium for evaluating HDR video quality based on residual timing correction. Background Technology

[0002] With the rapid development of the ultra-high-definition video industry, High Dynamic Range (HDR) video, with its wider brightness range, broader color gamut, and higher quantization bit depth, can present image effects that more closely resemble the real perception of the human eye, and has been widely used in streaming media on demand, live webcasting, film and television production, and broadcasting. However, HDR video inevitably introduces various distortions during acquisition, encoding compression, transcoding distribution, and terminal playback. For example, quantization, prediction, and transform coding may produce spatial distortions such as block boundaries, ringing, texture attenuation, and local structural damage, while temporal distortions such as flicker, cross-frame quality fluctuations, motion artifacts, and coding error propagation may also occur. To provide reliable data for services such as encoding parameter optimization, bitrate-quality control, bitstream quality inspection, and codec scheme evaluation, accurate evaluation of the perceived quality of HDR compressed video is necessary. Existing no-reference video quality evaluation methods still have significant shortcomings when dealing with HDR compressed video. On the one hand, HDR compression distortion is heterogeneous in spatial scale, with different distortions differing in spatial frequency and range of effect. A single observation scale is insufficient to simultaneously characterize local texture degradation and structural anomalies within a large receptive field. Extracting features solely from the original scale may cause the network to be biased towards content semantics or single-band responses, making it difficult to form a quality representation with scale complementarity. On the other hand, phenomena such as flicker, cross-frame quality fluctuations, motion region artifacts, and coding error propagation all depend on the relationship between consecutive frames. Frame-by-frame independent evaluation and mean pooling are insufficient to describe the aforementioned temporal instabilities and may weaken the impact of short-term significant distortions on overall quality. Furthermore, retraining the entire network directly in an end-to-end manner may not only alter the existing spatial quality scale but also significantly increase the risk of overfitting under limited HDR subjective quality data, making it difficult to guarantee the accuracy and stability of the evaluation results.

[0003] In conclusion, improving the accuracy and stability of HDR video quality evaluation is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for HDR video quality evaluation based on residual timing correction, which can improve the accuracy and stability of HDR video quality evaluation. The specific solution is as follows: In a first aspect, this application discloses an HDR video quality evaluation method based on residual temporal correction, including: Acquire training video data, construct a first training set based on the training video data, and construct a target view corresponding to each video frame in the first training set. Use an initial evaluator to obtain a first frame-by-frame basic quality response corresponding to the training video data based on the target view, and update the initial evaluator based on the first frame-by-frame basic quality response and the first subjective quality score corresponding to the training video data to obtain a target evaluator. The target view includes the original scale view corresponding to the video frame and at least one controlled resampling view. The initial evaluator is a pre-constructed multi-scale spatial basic evaluator. HDR consecutive frame segments are acquired, and a second training set is constructed using the HDR consecutive frame segments. The target evaluator is loaded into the initial video quality evaluation model, and the second frame-by-frame basic quality response, frame-by-frame spatial latent feature, and frame-by-frame spatiotemporal latent feature corresponding to each HDR consecutive frame segment in the second training set are obtained using the initial video quality evaluation model. The HDR consecutive frame segments carry a corresponding second subjective quality score. Using the residual prediction head of the initial video quality evaluation model, the frame-by-frame spatial latent features and the frame-by-frame spatiotemporal latent features corresponding to each other in the temporal position of the HDR continuous frame segment are fused to obtain the target joint features, and the frame-by-frame residual correction amount corresponding to the HDR continuous frame segment is determined based on the target joint features. The training quality score corresponding to the HDR continuous frame segment is obtained based on the second frame-by-frame basic quality response and the frame-by-frame residual correction. The initial video quality evaluation model is updated based on the training quality score and the second subjective quality score to obtain the target video quality evaluation model. The target video quality evaluation model is then used to obtain the target video quality evaluation result corresponding to the HDR video to be evaluated.

[0005] Optionally, obtaining the first frame-by-frame basic quality response corresponding to the training video data based on the target view using the initial evaluator includes: The target feature extraction network of the initial evaluator is used to extract features from the target view corresponding to each video frame of the training video data to obtain the corresponding quality-aware vector, and the quality-aware vectors corresponding to each target view of the video frame are connected to obtain the target connection features corresponding to the video frame. The first frame-by-frame basic quality response corresponding to the training video data is determined based on the target connectivity features corresponding to each video frame using the multi-scale feature fusion layer and basic quality mapping layer of the initial evaluator.

[0006] Optionally, updating the initial evaluator based on the first frame-by-frame basic quality response and the first subjective quality score corresponding to the training video data to obtain the target evaluator includes: Based on the first frame-by-frame basic quality response and the first subjective quality score corresponding to the training video data, the first quality prediction loss corresponding to the first training set is determined, and the initial evaluator is updated based on the first quality prediction loss to obtain the updated initial evaluator. The updated initial evaluator is used to obtain the evaluation result of the first correlation index corresponding to the first preset HDR validation set, and when the evaluation result of the first correlation index meets the first iteration termination condition, the updated initial evaluator is determined as the target evaluator.

[0007] Optionally, obtaining the second frame-by-frame basic quality response, frame-by-frame spatial latent features, and frame-by-frame spatiotemporal latent features corresponding to each of the HDR consecutive frame segments in the second training set using the initial video quality evaluation model includes: The target evaluator in the initial video quality evaluation model is used to obtain the second frame-by-frame basic quality response and frame-by-frame spatial latent features corresponding to each HDR continuous frame segment in the second training set. The spatiotemporal feature extraction branch of the initial video quality evaluation model is used to obtain the frame-by-frame spatiotemporal latent features corresponding to each HDR continuous frame segment in the second training set.

[0008] Optionally, obtaining the training quality score corresponding to the HDR consecutive frame segment based on the second frame-by-frame basic quality response and the frame-by-frame residual correction includes: The second frame-by-frame basic quality response is time-aggregated to obtain a first aggregation result, and the frame-by-frame residual correction is time-aggregated to obtain a second aggregation result; The training quality score corresponding to the HDR consecutive frame segment is determined based on the first aggregation result and the second aggregation result.

[0009] Optionally, updating the initial video quality assessment model based on the training quality score and the second subjective quality score to obtain the target video quality assessment model includes: Based on the training quality score and the second subjective quality score corresponding to each HDR consecutive frame segment in the second training set, a target relevance constraint and a target ranking constraint are determined, and a second quality prediction loss corresponding to the second training set is determined based on the target relevance constraint and the target ranking constraint. Based on the second quality prediction loss, the spatiotemporal feature extraction branch of the initial video quality assessment model and the residual prediction head are updated to obtain the updated initial video quality assessment model. The updated initial video quality evaluation model is used to obtain the evaluation result of the second correlation index corresponding to the second preset HDR validation set. When the evaluation result of the second correlation index meets the second iteration termination condition, the updated initial video quality evaluation model is determined as the target video quality evaluation model.

[0010] Optionally, obtaining the target video quality evaluation result corresponding to the HDR video to be evaluated using the target video quality evaluation model includes: The HDR video to be evaluated is acquired, and the HDR video to be evaluated is decoded, the encoded value is normalized, the spatial size is regularized, and deterministic temporal sampling is performed to obtain the continuous frame segments to be evaluated corresponding to the HDR video to be evaluated; the continuous frame segments to be evaluated include at least one continuous frame segment of the HDR video to be evaluated. The continuous frame segments to be evaluated are input into the target video quality evaluation model to obtain the target frame-by-frame basic quality response and the target frame-by-frame residual correction amount corresponding to the continuous frame segments to be evaluated, and the segment-level quality score corresponding to the continuous frame segments to be evaluated is determined based on the target frame-by-frame basic quality response and the target frame-by-frame residual correction amount. The target video quality evaluation result is determined based on the segment-level quality score corresponding to the consecutive frame segments of the HDR video to be evaluated.

[0011] Secondly, this application discloses an HDR video quality evaluation device based on residual timing correction, comprising: An evaluator training module is used to acquire training video data, construct a first training set based on the training video data, and build a target view corresponding to each video frame in the first training set. An initial evaluator is used to acquire a first frame-by-frame basic quality response corresponding to the training video data based on the target view. The initial evaluator is then updated based on the first frame-by-frame basic quality response and a first subjective quality score corresponding to the training video data to obtain a target evaluator. The target view includes the original scale view corresponding to the video frame and at least one controlled resampling view. The initial evaluator is a pre-constructed multi-scale spatial basic evaluator. The training feature extraction module is used to acquire HDR continuous frame segments, construct a second training set using the HDR continuous frame segments, load the target evaluator into the initial video quality evaluation model, and use the initial video quality evaluation model to acquire the second frame-by-frame basic quality response, frame-by-frame spatial latent features, and frame-by-frame spatiotemporal latent features corresponding to each HDR continuous frame segment in the second training set; the HDR continuous frame segments carry corresponding second subjective quality scores. The residual correction amount determination module is used to use the residual prediction head of the initial video quality evaluation model to fuse the frame-by-frame spatial latent features and the frame-by-frame spatiotemporal latent features that correspond to each other in the temporal position of the HDR continuous frame segment to obtain the target joint features, and to determine the frame-by-frame residual correction amount corresponding to the HDR continuous frame segment based on the target joint features. The video quality evaluation module is used to obtain the training quality score corresponding to the HDR continuous frame segment based on the second frame-by-frame basic quality response and the frame-by-frame residual correction amount, update the initial video quality evaluation model based on the training quality score and the second subjective quality score to obtain the target video quality evaluation model, and use the target video quality evaluation model to obtain the target video quality evaluation result corresponding to the HDR video to be evaluated.

[0012] Thirdly, this application discloses an electronic device, including: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned HDR video quality evaluation method based on residual timing correction.

[0013] Fourthly, this application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned HDR video quality evaluation method based on residual timing correction.

[0014] In this application, when evaluating the video quality of HDR videos based on residual temporal correction, training video data is acquired, a first training set is constructed based on the training video data, and a target view corresponding to each video frame in the first training set is constructed. An initial evaluator is used to obtain a first frame-by-frame basic quality response corresponding to the training video data based on the target view, and the initial evaluator is updated based on the first frame-by-frame basic quality response and the first subjective quality score corresponding to the training video data to obtain a target evaluator. The target view includes the original scale view corresponding to the video frame and at least one controlled resampling view. The initial evaluator is a pre-constructed multi-scale spatial basic evaluator. HDR continuous frame segments are acquired, a second training set is constructed using the HDR continuous frame segments, the target evaluator is loaded into the initial video quality evaluation model, and the target evaluator is used to obtain the first frame-by-frame basic quality response corresponding to each video frame in the second training set. The HDR continuous frame segment contains a second frame-by-frame basic quality response, frame-by-frame spatial latent features, and frame-by-frame spatiotemporal latent features. Each HDR continuous frame segment carries a corresponding second subjective quality score. Using the residual prediction head of the initial video quality evaluation model, the frame-by-frame spatial latent features and frame-by-frame spatiotemporal latent features corresponding to each other in temporal positions within the HDR continuous frame segment are fused to obtain target joint features. Based on the target joint features, the frame-by-frame residual correction amount corresponding to the HDR continuous frame segment is determined. Based on the second frame-by-frame basic quality response and the frame-by-frame residual correction amount, the training quality score corresponding to the HDR continuous frame segment is obtained. Based on the training quality score and the second subjective quality score, the initial video quality evaluation model is updated to obtain a target video quality evaluation model. Finally, the target video quality evaluation model is used to obtain the target video quality evaluation result corresponding to the HDR video to be evaluated. As can be seen, this application adopts a phased optimization approach. First, it trains a multi-scale spatial base evaluator using a target view that includes the original scale view and at least one controlled resampled view, enabling the base quality evaluation to simultaneously utilize the distortion-sensitive response at different observation scales. Then, it freezes the multi-scale spatial base evaluator and uses a spatiotemporal feature extraction branch and a residual prediction head to fuse the frame-by-frame spatial latent features and frame-by-frame spatiotemporal latent features corresponding to the temporal positions and estimate the frame-by-frame residual correction. The quality score is obtained by additively fusing the temporal aggregation results of the base quality response and the temporal aggregation results of the residual correction. This approach not only supplements cross-frame variation information but also maintains the existing spatial quality scale, reducing the optimization instability caused by the competition between the base quality evaluation and the temporal residual correction, thereby improving the accuracy and stability of HDR video quality evaluation. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0016] Figure 1 This is a flowchart of an HDR video quality evaluation method based on residual temporal correction disclosed in this application; Figure 2 This is a schematic diagram illustrating the training process of a specific HDR video quality assessment model disclosed in this application; Figure 3 This is a schematic diagram illustrating the phased training process of a specific HDR video quality assessment model disclosed in this application; Figure 4 This is a schematic diagram of a specific HDR video quality evaluation process disclosed in this application; Figure 5 This is a schematic diagram of the structure of an HDR video quality evaluation device based on residual timing correction disclosed in this application; Figure 6 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Existing no-reference video quality assessment methods still have significant shortcomings when dealing with HDR compressed videos. On the one hand, HDR compression distortion is heterogeneous in spatial scale, with different distortions differing in spatial frequency and range of effect. A single observation scale cannot simultaneously characterize local texture degradation and structural anomalies under a large receptive field. Extracting features only from the original scale may cause the network to be biased towards content semantics or single-band response, making it difficult to form a quality representation with scale complementarity. On the other hand, phenomena such as flicker, cross-frame quality fluctuations, motion region artifacts, and coding error propagation all depend on the relationship between consecutive frames. Frame-by-frame independent evaluation and mean pooling are insufficient to describe the above-mentioned temporal instabilities and may weaken the impact of short-term significant distortions on overall quality. Furthermore, directly retraining the entire network in an end-to-end manner may not only change the existing spatial quality scale but also significantly increase the risk of overfitting under limited HDR subjective quality data, making it difficult to guarantee the accuracy and stability of the evaluation results. To address the above technical problems, this application discloses an HDR video quality assessment method based on residual temporal correction, which can improve the accuracy and stability of HDR video quality assessment.

[0019] See Figure 1 As shown, this embodiment of the invention discloses a method for evaluating HDR video quality based on residual temporal correction, including: Step S11: Obtain training video data, construct a first training set based on the training video data, and construct a target view corresponding to each video frame in the first training set. Use an initial evaluator to obtain a first frame-by-frame basic quality response corresponding to the training video data based on the target view, and update the initial evaluator based on the first frame-by-frame basic quality response and the first subjective quality score corresponding to the training video data to obtain a target evaluator. The target view includes the original scale view corresponding to the video frame and at least one controlled resampling view. The initial evaluator is a pre-constructed multi-scale spatial basic evaluator.

[0020] In this embodiment, as Figure 2The diagram illustrates the training process of a specific HDR video quality assessment model. The training video data consists of video data carrying a first subjective quality score, which can be composed of one or more video quality databases with subjective quality scores. This database can include HDR videos as well as videos with other dynamic ranges. The first subjective quality score can be an average opinion score obtained through subjective viewing experiments, or its corresponding difference average opinion score, or other scores that characterize the subjective perceived quality of the video. During the construction of the first training set based on the training video data, videos of different resolutions, frame rates, and durations can be decoded, encoded, normalized, spatially regularized, and temporally sampled to form a normalized frame tensor. The first training stage can use either single-frame samples to train the initial evaluator or multiple-frame samples for training, with each video frame inheriting the first subjective quality score corresponding to its respective video.

[0021] In one specific implementation, the first training set can consist of multiple video quality databases with subjective quality scores, and employ homogeneous batch sampling, meaning that samples within the same training batch come from the same database. This reduces optimization interference caused by the mixing of different database subdivisions and input statistics within the same batch. For example, one frame can be randomly selected from each video as a training sample, and the input spatial size can be normalized to 512×512. When constructing the target view, the scale factor set can be set to {1,4}, where scale factor 1 corresponds to the original scale view, and scale factor 4 corresponds to the controlled resampling view. Nearest neighbor interpolation is used to first reduce the video frame to one-quarter of its original spatial size, and then restore it to its original spatial size. It is understood that the above-described number of scales, scale factor values, and interpolation methods are only used to illustrate one possible implementation of this application and do not constitute a limitation on this embodiment.

[0022] It should be noted that the target view includes the original scale view corresponding to the video frame and at least one controlled resampling view. Specifically, a set of scale factors R={r_1,…,r_K} can be pre-defined, where at least one scale factor corresponds to the original scale and at least one scale factor corresponds to the controlled resampling scale. The original scale view maintains the original spatial sampling structure of the input. For the controlled resampling view corresponding to the controlled resampling scale, the spatial size of the video frame can be reduced by the corresponding scale factor and then restored to the original spatial size. Alternatively, other resampling methods that change the observation scale while maintaining a consistent output size can be used. It is understood that the resampling ratio, interpolation method, and number of scales can all be configured according to the application scenario. In this way, by using resampling operations to change the local sampling structure and the presentation of high-frequency information, the original scale view retains fine-grained texture and local artifact information, while the controlled resampling view changes the spatial sampling structure, enabling the model to obtain complementary compression distortion evidence from different observation scales, forming a complementary distortion response.

[0023] In this embodiment, the initial evaluator is a pre-constructed multi-scale spatial base evaluator, which may include a target feature extraction network, a multi-scale feature fusion layer, and a base quality mapping layer. The target feature extraction network is a quality-aware feature extraction network with shared parameters for each target view within the same video frame. Sharing the same quality-aware feature extraction parameters across different scales avoids the generation of unconstrained feature spaces by independent feature extraction networks, improves the comparability of cross-scale features, and facilitates the comparison and combination of responses at different scales by the multi-scale feature fusion layer. It is understood that the type, hidden layer dimension, and number of multi-scale feature fusion layers of the quality-aware feature extraction network do not constitute a limitation on this application. For example, the target feature extraction network may employ a convolutional neural network or other network structures with quality-aware feature extraction capabilities. After obtaining the first frame-by-frame base quality response based on the target view using the initial evaluator, the corresponding quality prediction loss can be calculated based on the first frame-by-frame base quality response and the first subjective quality score. This loss is then used to perform supervised optimization of the initial evaluator. Furthermore, the parameters of the multi-scale spatial base evaluator can be selected based on preset correlation indicators on an independent HDR validation set, thereby determining the updated initial evaluator that meets the conditions as the target evaluator.

[0024] In this embodiment, the initial evaluator is used to obtain the first frame-by-frame basic quality response corresponding to the training video data based on the target view. Specifically, this may include: using the target feature extraction network of the initial evaluator to extract features from the target view corresponding to each video frame of the training video data to obtain the corresponding quality-aware vector, and concatenating the quality-aware vectors corresponding to each target view of the video frame to obtain the target connection features corresponding to the video frame; and using the multi-scale feature fusion layer and basic quality mapping layer of the initial evaluator to determine the first frame-by-frame basic quality response corresponding to the training video data based on the target connection features corresponding to each video frame.

[0025] In one specific implementation, the target feature extraction network can use EfficientNetV2-S as a shared quality-aware feature extraction network, and the target view at each scale of the video frame is mapped into a 128-dimensional scale-level quality-aware vector by this network. The quality-perceived vectors corresponding to each scale are concatenated along the feature dimension to obtain the target connectivity features corresponding to the video frame. Then, the target connectivity features are processed by a multi-scale feature fusion layer and a basic quality mapping layer to obtain frame-by-frame spatial latent features of a preset dimension and a frame-by-frame basic quality response in scalar form, for example, obtaining 128-dimensional frame-by-frame spatial latent features. In this way, feature-level concatenation is performed first after the features at each scale are formed, and then the basic quality response is uniformly regressed. This avoids prematurely compressing the information at each scale into independent scalar scores before scale fusion, thus preserving scale-level quality evidence.

[0026] In this embodiment, as Figure 3 As shown, the training process of the HDR video quality assessment model can be divided into two stages. The first stage is to train the initial evaluator, and the second stage is to train the initial video quality assessment model. Specifically, the initial evaluator is updated based on the first frame-by-frame basic quality response and the first subjective quality score corresponding to the training video data to obtain the target evaluator. This can include: determining the first quality prediction loss corresponding to the first training set based on the first frame-by-frame basic quality response and the first subjective quality score corresponding to the training video data, and updating the initial evaluator based on the first quality prediction loss to obtain the updated initial evaluator; using the updated initial evaluator to obtain the evaluation result of the first correlation index corresponding to the first preset HDR validation set, and determining the updated initial evaluator as the target evaluator when the evaluation result of the first correlation index meets the first iteration termination condition.

[0027] In one specific implementation, let the predicted quality score be q and the subjective quality score be y, then the first quality prediction loss... It can be represented as: ; in, For correlation constraints, Pearson linear correlation coefficient loss can be used, for example. For sorting constraints, pairwise sorting loss can be used, for example. The loss weights are greater than zero. In one specific implementation, the Adam optimizer can be used in the first training phase, with a training batch size of 16 and a learning rate of... The training process was conducted for 45 rounds. The loss function was the sum of Pearson linear correlation coefficient loss and pairwise ranking loss, with a ranking loss weight of 5.

[0028] Furthermore, deterministic continuous frame segment sampling can be employed on an independent first preset HDR validation set, and the evaluation result of the first correlation index can be determined based on the Spearman rank correlation coefficient and the Pearson linear correlation coefficient. Correspondingly, the first iteration termination condition can be that the evaluation result of the first correlation index reaches a preset index threshold, no further improvement is achieved within a preset number of rounds, or the training reaches a preset maximum number of rounds. When the evaluation result of the first correlation index meets the first iteration termination condition, the updated initial evaluator with the optimal correlation index on the validation set is determined as the target evaluator, that is, the multi-scale spatial base evaluator parameters are selected based on the preset correlation index on the independent HDR validation set.

[0029] Step S12: Obtain HDR continuous frame segments, construct a second training set using the HDR continuous frame segments, load the target evaluator into the initial video quality evaluation model, and use the initial video quality evaluation model to obtain the second frame-by-frame basic quality response, frame-by-frame spatial latent features, and frame-by-frame spatiotemporal latent features corresponding to each HDR continuous frame segment in the second training set; the HDR continuous frame segments carry corresponding second subjective quality scores.

[0030] In this embodiment, an HDR compressed distortion video stream with a subjective quality score can be acquired, and then decoded, encoded, normalized, spatially regularized, and temporally sampled. Continuous frame segments are then constructed according to a preset segment length to form a second training set. During training, HDR continuous frame segments can be randomly selected from different starting positions to increase temporal coverage. The normalized continuous frame segment can be represented as X=[B,F,C,H,W], where B represents the batch size, F represents the number of segments and frames, C represents the number of channels, and H and W represent the spatial height and width, respectively. Each HDR continuous frame segment carries a corresponding second subjective quality score, which is the subjective quality score of the HDR video to which the HDR continuous frame segment belongs.

[0031] In one specific implementation, the training data for the second training phase uses only HDR videos with subjective quality scores; each training sample is a continuous frame segment containing 10 frames, truncated with a random starting offset, and the input space size is normalized to 512×512. The second training phase loads the parameters of the multi-scale spatial base evaluator selected in the first training phase and freezes the quality-aware feature extraction network, multi-scale feature fusion layer, base quality mapping layer, and batch normalization statistics of the multi-scale spatial base evaluator. Furthermore, it does not construct a parameter backpropagation graph for the multi-scale spatial base evaluator. For other front-end modules that do not participate in data processing in the second training phase, their forward computation is disabled.

[0032] Understandably, the initial video quality assessment model includes a target evaluator, a spatiotemporal feature extraction branch, and a residual prediction head. Loading the target evaluator into the initial video quality assessment model involves loading the multi-scale spatial base evaluator parameters obtained in the first stage of training. In one specific implementation, the target feature extraction network, multi-scale feature fusion layer, base quality mapping layer, and their normalized statistics of the target evaluator can be frozen, and the construction of the parameter backpropagation graph can be stopped during the forward pass of the target evaluator. For other front-end modules that do not participate in data processing in the second training stage, their forward pass can also be turned off to reduce computational overhead.

[0033] In this embodiment, the second frame-by-frame basic quality response, frame-by-frame spatial latent features, and frame-by-frame spatiotemporal latent features corresponding to each HDR continuous frame segment in the second training set are obtained using the initial video quality evaluation model. Specifically, this may include: obtaining the second frame-by-frame basic quality response and frame-by-frame spatial latent features corresponding to each HDR continuous frame segment in the second training set using the target evaluator in the initial video quality evaluation model; and obtaining the frame-by-frame spatiotemporal latent features corresponding to each HDR continuous frame segment in the second training set using the spatiotemporal feature extraction branch of the initial video quality evaluation model.

[0034] When acquiring features, on the one hand, the target evaluator in the initial video quality assessment model is used to process the HDR continuous frame segments frame by frame. That is, a target view containing the original scale view and the controlled resampling view is constructed for each video frame in the segment and features are extracted to obtain the second frame-by-frame basic quality response S_base=[B,F] and frame-by-frame spatial latent features H=[B,F,D], where D is the preset dimension of the frame-by-frame spatial latent features. On the other hand, the spatiotemporal feature extraction branch of the initial video quality assessment model is used to obtain the frame-by-frame spatiotemporal latent features V=[B,F,C_v], where C_v is the channel dimension of the frame-by-frame spatiotemporal latent features.

[0035] Specifically, the spatiotemporal feature extraction branch can first rearrange the continuous HDR frame segments from [B,F,C,H,W] to [B,C,F,H,W], and normalize the spatial dimensions to a preset size while maintaining the time length of F. The spatiotemporal feature extraction branch includes a 3D convolutional initiation layer and at least one 3D convolutional coding unit. Each 3D convolution uses a stride and boundary padding method that maintains the number of time positions in the temporal direction, and extracts local structural information step by step in the spatial direction. Then, the output of the 3D convolution is aggregated only along the spatial dimension, and the dimension is restored to [B,F,C_v] to obtain the frame-by-frame spatiotemporal latent features corresponding to the positions of each input frame. Furthermore, a normalization layer can be used to stabilize feature statistics. It is understandable that the number of 3D convolutional layers, the spatial normalization size, the channel width, and the normalization method can all be configured as needed.

[0036] It should be noted that the spatiotemporal feature extraction branch and the target evaluator receive the same continuous HDR frame segments. They use the same frame index and time order, but they do not share parameters. This ensures that the frame-by-frame spatial latent features and the frame-by-frame spatiotemporal latent features can correspond one-to-one according to the time position, providing an alignment basis for subsequent feature fusion.

[0037] In one specific implementation, when acquiring frame-by-frame spatiotemporal latent features, the spatiotemporal feature extraction branch can first use adaptive average pooling to normalize the spatial size to 256×256 while maintaining the temporal length at 10. Subsequently, a three-dimensional convolutional initiation layer and a three-level three-dimensional convolutional coding unit are used. After each three-dimensional convolution, three-dimensional batch normalization and GELU activation are set, and the stride of each three-dimensional convolution in the temporal direction is 1 to keep the number of temporal positions unchanged. The hidden dimension parameter of the spatiotemporal feature extraction branch can be set to 128, and the channel compression coefficient can be set to 8. After aggregating the three-dimensional convolution output along only the spatial dimension and restoring the dimension, a 32-dimensional frame-by-frame spatiotemporal latent feature is finally obtained.

[0038] Step S13: Using the residual prediction head of the initial video quality evaluation model, fuse the frame-by-frame spatial latent features and the frame-by-frame spatiotemporal latent features that correspond to each other in the temporal positions of the HDR continuous frame segment to obtain the target joint features, and determine the frame-by-frame residual correction amount corresponding to the HDR continuous frame segment based on the target joint features.

[0039] In this embodiment, for each time position f in a continuous HDR frame segment, the frame-by-frame spatial latent feature corresponding to that time position is... With frame-by-frame spatiotemporal latent features Connect along the feature dimension to construct joint features of the target: ; in, Indicates to and The connections are made along the feature dimension. Then, the joint target features are input into the residual prediction head to obtain the frame-by-frame residual correction. In one specific implementation, the residual prediction head can adopt a "linear mapping-GELU-linear mapping" structure, that is, the joint features of the target are first transformed through the first linear mapping layer, then nonlinearly processed by the GELU activation function, and finally the frame-by-frame residual correction amount is obtained by regression through the second linear mapping layer.

[0040] Understandably, the residual prediction head simultaneously receives frame-by-frame spatial latent features and frame-by-frame spatiotemporal latent features, making the frame-by-frame residual correction amount constrained by both the spatial quality state of the current frame and the local cross-frame changes, thus forming content-related temporal corrections. Furthermore, the joint encoding of adjacent frames by 3D convolution while maintaining the temporal position correspondence can provide additional correction basis for phenomena that are difficult to describe in frame-by-frame basic quality evaluation, such as flicker, cross-frame quality fluctuations, motion region artifacts, and error propagation, and supplement cross-frame change information.

[0041] Step S14: Obtain the training quality score corresponding to the HDR continuous frame segment based on the second frame-by-frame basic quality response and the frame-by-frame residual correction amount; update the initial video quality evaluation model based on the training quality score and the second subjective quality score to obtain the target video quality evaluation model; and use the target video quality evaluation model to obtain the target video quality evaluation result corresponding to the HDR video to be evaluated.

[0042] In this embodiment, the base quality score given by the frozen multi-scale spatial base evaluator is used as the quality prior. The residual bias in the target subjective quality scale is estimated using frame-by-frame spatiotemporal latent features, and video-level correction is completed using additive residuals. In a specific implementation, the training quality score corresponding to the HDR continuous frame segment is obtained based on the second frame-by-frame base quality response and the frame-by-frame residual correction amount, including: performing temporal aggregation on the second frame-by-frame base quality response to obtain a first aggregation result, and performing temporal aggregation on the frame-by-frame residual correction amount to obtain a second aggregation result; and determining the training quality score corresponding to the HDR continuous frame segment based on the first aggregation result and the second aggregation result.

[0043] Specifically, when obtaining the training quality score, the second frame-by-frame basic quality response can be temporally aggregated to obtain the first aggregation result. The frame-by-frame residual corrections are then aggregated over time to obtain a second aggregation result. ,Right now: ; ; in, This represents the second frame-by-frame basic quality response corresponding to frame f. This represents the frame-by-frame residual correction amount corresponding to the f-th frame, where F represents the number of frames in the HDR consecutive frame segment. Then, the first aggregation result is added to the second aggregation result to obtain the training quality score corresponding to the HDR consecutive frame segment. ,Right now: ; in, This represents the basic quality score. This represents the video-level residual correction amount determined jointly by frame-by-frame spatial latent features and frame-by-frame spatiotemporal latent features. It should be noted that this application does not use the frame-by-frame residual correction amount as a multiplicative weight to rewrite the second frame-by-frame basic quality response. Instead, it employs an additive aggregation of the basic quality score and the video-level residual correction amount, thereby explicitly separating the functional boundaries of basic quality evaluation and temporal residual correction. The basic quality score remains independent, preserving the basic quality anchor point. Furthermore, the frame-by-frame residual correction amount... It can also be used as an auxiliary quantity to analyze the correction contribution at different time locations.

[0044] In this embodiment, updating the initial video quality evaluation model based on the training quality score and the second subjective quality score to obtain the target video quality evaluation model includes: determining the target relevance constraint term and the target ranking constraint term based on the training quality score and the second subjective quality score corresponding to each HDR consecutive frame segment in the second training set, and determining the second quality prediction loss corresponding to the second training set based on the target relevance constraint term and the target ranking constraint term; updating the spatiotemporal feature extraction branch and residual prediction head of the initial video quality evaluation model based on the second quality prediction loss to obtain the updated initial video quality evaluation model; using the updated initial video quality evaluation model to obtain the second relevance index evaluation result corresponding to the second preset HDR validation set, and determining the updated initial video quality evaluation model as the target video quality evaluation model when the second relevance index evaluation result meets the second iteration termination condition.

[0045] In this embodiment, the second training phase can employ the same form of joint correlation and ranking loss as the first training phase. Specifically, the target correlation constraint term can be a loss term constructed based on the Pearson linear correlation coefficient, for example: ; Where PLCC(S,y) represents the Pearson linear correlation coefficient between the training quality score S and the second subjective quality score y corresponding to each consecutive HDR frame segment within the training batch. The target ranking constraint term can be a pairwise ranking loss, for example: ; The summation iterates through sample pairs (i,j) within the training batch. and These represent the training quality scores corresponding to the i-th and j-th consecutive HDR frame segments, respectively. and Let represent the corresponding second subjective quality scores, sign(·) be the sign function, and m be the preset interval. The second quality prediction loss is a weighted sum of the target relevance constraint and the target ranking constraint. In one specific implementation, the second training phase can use the Adam optimizer with a training batch size of 16 and a learning rate of . The training process was conducted for 45 rounds, with a ranking loss weight of 5, and only the parameters of the spatiotemporal feature extraction branch and the residual prediction head were updated.

[0046] Furthermore, the updated initial video quality evaluation model is validated on an independent second preset HDR validation set. The evaluation result of the second correlation index is determined based on the Spearman rank correlation coefficient and the Pearson linear correlation coefficient. Correspondingly, the second iteration termination condition can be that the evaluation result of the second correlation index reaches a preset index threshold, no longer improves within a preset number of rounds, or the training reaches a preset maximum number of rounds. When the evaluation result of the second correlation index meets the second iteration termination condition, the updated initial video quality evaluation model is determined as the target video quality evaluation model, that is, the final model parameters are selected based on the preset correlation index on the independent HDR validation set.

[0047] In one specific implementation, after obtaining the training quality score, the corresponding quality prediction loss is calculated based on the training quality score and the second subjective quality score. Only the spatiotemporal feature extraction branch and the residual prediction head of the initial video quality assessment model are updated. The frozen target evaluator provides a stable basic quality anchor point. The spatiotemporal feature extraction branch and the residual prediction head only learn frame-by-frame residual corrections that cannot be fully explained by independent frames, without needing to relearn the complete quality mapping. This reduces the difficulty of residual learning and decreases the size of trainable parameters when optimizing with limited HDR subjective quality data, mitigating overfitting under limited HDR data. Furthermore, the final model parameters can be selected based on a preset correlation index on an independent HDR validation set, thus obtaining the target video quality assessment model.

[0048] In this embodiment, the target video quality evaluation result corresponding to the HDR video to be evaluated is obtained using a target video quality evaluation model. This includes: acquiring the HDR video to be evaluated, and decoding, encoding value normalization, spatial size regularization, and deterministic temporal sampling of the HDR video to be evaluated to obtain the continuous frame segments to be evaluated corresponding to the HDR video to be evaluated; the continuous frame segments to be evaluated include at least one continuous frame segment of the HDR video to be evaluated; inputting the continuous frame segments to be evaluated into the target video quality evaluation model to obtain the target frame-by-frame basic quality response and target frame-by-frame residual correction amount corresponding to the continuous frame segments to be evaluated, and determining the segment-level quality score corresponding to the continuous frame segments to be evaluated based on the target frame-by-frame basic quality response and target frame-by-frame residual correction amount; and determining the target video quality evaluation result corresponding to the HDR video to be evaluated based on the segment-level quality score corresponding to the continuous frame segments to be evaluated of the HDR video to be evaluated.

[0049] In other words, after training is complete, this embodiment performs the same preprocessing as the training phase on the HDR video to be evaluated to obtain continuous frame segments for evaluation. These segments are then input into the target video quality evaluation model. Based on the target frame-by-frame basic quality response and target frame-by-frame residual correction output by the model, the segment-level quality score is determined according to the aforementioned additive residual aggregation relationship, thereby determining the target video quality evaluation result corresponding to the HDR video to be evaluated. It is understood that deterministic temporal sampling can be used in the verification and quality evaluation phases to ensure the repeatability of the evaluation results.

[0050] In one specific implementation, such as Figure 4 As shown, the quality evaluation stage first acquires the HDR compressed video to be evaluated, performs decoding, encoding value normalization, spatial size regularization, and deterministic temporal sampling to obtain at least one continuous frame segment to be evaluated; deterministic temporal sampling ensures the repeatability of the evaluation results. Then, a multi-scale spatial view containing the original scale view and the controlled resampled view is constructed for the video frames in each continuous frame segment to be evaluated. The trained target evaluator obtains the target frame-by-frame basic quality response and frame-by-frame spatial latent features. Simultaneously, the same continuous frame segments to be evaluated are input into the spatiotemporal feature extraction branch to obtain frame-by-frame spatiotemporal latent features aligned with the temporal position of the frame-by-frame spatial latent features. Next, the frame-by-frame spatial latent features and frame-by-frame spatiotemporal latent features at corresponding temporal positions are connected, and the target frame-by-frame residual correction amount is obtained through the residual prediction head. Finally, the segment-level quality score is obtained according to the additive residual aggregation relationship. ; in, This indicates that the average value is taken along the time position f. This represents the target's frame-by-frame basic quality response. This represents the target frame-by-frame residual correction amount. When the HDR video to be evaluated contains multiple consecutive frame segments to be evaluated, the segment dimension is preserved and each segment-level quality score is calculated separately. Then, a preset aggregation is performed on each of the segment-level quality scores to obtain the video-level no-reference quality evaluation result, which is the target video quality evaluation result corresponding to the HDR video to be evaluated.

[0051] In one specific implementation, during verification and quality evaluation, 20 consecutive frame segments, each containing 20 frames, can be deterministically selected from each HDR video; a segment-level quality score is then calculated for each consecutive frame segment according to the formula described above. Then, for 20 consecutive frame segments The arithmetic mean is taken to obtain the video-level quality score. It is understood that this continuous frame segment-level averaging method is consistent with the verification procedure in this embodiment; the number of continuous frame segments, the length of continuous frame segments, and the video-level aggregation method can all be adjusted according to the duration of the video to be evaluated and application requirements.

[0052] Furthermore, the target video quality evaluation results obtained in this embodiment can be applied to scenarios such as encoding parameter optimization, bitrate-quality control, bitstream quality inspection, and codec scheme evaluation. For example, the quantization parameters or target bitrate of the encoder can be adjusted based on the target video quality evaluation results to reduce transmission bandwidth while ensuring perceived quality. Another example is the periodic quality inspection of HDR bitstreams in the distribution link, triggering an alarm when the target video quality evaluation result falls below a preset threshold. Yet another example is the use of the target video quality evaluation results to objectively compare the perceived quality of different codec schemes under the same bitrate conditions, providing a basis for codec scheme selection.

[0053] As can be seen, the multi-scale spatial base evaluator in this application constructs complementary observation views through controlled scale perturbation and extracts comparable quality-perceived representations using a shared quality-perceived feature extraction network. Firstly, it can form complementary distortion responses, i.e., the original scale view retains fine-grained texture and local artifact information, while the controlled resampling view changes the spatial sampling structure, enabling the model to obtain complementary compressed distortion evidence from different observation scales. Secondly, it can improve the comparability of cross-scale features, i.e., each scale view shares the same quality-perceived feature extraction parameters, avoiding the generation of unconstrained feature spaces by independent feature extraction networks. Thirdly, it can preserve scale-level quality evidence, i.e., feature-level connections are performed after the features at each scale are formed, and then the base quality response is uniformly regressed. Meanwhile, this embodiment performs residual temporal correction under the condition of frozen backbone, decoupling video quality prediction into two complementary subtasks: basic quality assessment and temporal residual correction. First, it can supplement cross-frame variation information, providing additional correction basis for phenomena such as flicker, cross-frame quality fluctuations, motion region artifacts, and error propagation. Second, it can reduce the difficulty of residual learning, so that the spatiotemporal feature extraction branch and residual prediction head do not need to relearn the complete quality mapping, but instead correct the part that is not fully explained by the basic quality score. Third, it can alleviate overfitting under limited HDR data and reduce the size of trainable parameters. Fourth, it can form content-related temporal correction, so that the frame-by-frame residual correction amount is constrained by both the spatial quality state of the current frame and the local cross-frame variation. Fifth, it can maintain the basic quality anchor point, keep the basic quality score independent, and the video-level residual correction amount can be analyzed as a contribution to temporal correction, thereby improving the accuracy, stability, and interpretability of HDR video quality assessment as a whole.

[0054] See Figure 5 As shown, this application discloses an HDR video quality evaluation device based on residual timing correction, comprising: The evaluator training module 11 is used to acquire training video data, construct a first training set based on the training video data, and construct a target view corresponding to each video frame in the first training set. It uses an initial evaluator to acquire a first frame-by-frame basic quality response corresponding to the training video data based on the target view, and updates the initial evaluator based on the first frame-by-frame basic quality response and a first subjective quality score corresponding to the training video data to obtain a target evaluator. The target view includes the original scale view corresponding to the video frame and at least one controlled resampling view. The initial evaluator is a pre-constructed multi-scale spatial basic evaluator. The training feature extraction module 12 is used to acquire HDR continuous frame segments, construct a second training set using the HDR continuous frame segments, load the target evaluator into the initial video quality evaluation model, and use the initial video quality evaluation model to acquire the second frame-by-frame basic quality response, frame-by-frame spatial latent features, and frame-by-frame spatiotemporal latent features corresponding to each HDR continuous frame segment in the second training set; the HDR continuous frame segments carry corresponding second subjective quality scores. The residual correction amount determination module 13 is used to use the residual prediction head of the initial video quality evaluation model to fuse the frame-by-frame spatial latent features and the frame-by-frame spatiotemporal latent features that correspond to each other in the temporal position in the HDR continuous frame segment to obtain the target joint features, and to determine the frame-by-frame residual correction amount corresponding to the HDR continuous frame segment based on the target joint features. The video quality evaluation module 14 is used to obtain the training quality score corresponding to the HDR continuous frame segment based on the second frame-by-frame basic quality response and the frame-by-frame residual correction amount, update the initial video quality evaluation model based on the training quality score and the second subjective quality score to obtain the target video quality evaluation model, and use the target video quality evaluation model to obtain the target video quality evaluation result corresponding to the HDR video to be evaluated.

[0055] As can be seen, this application adopts a phased optimization approach. First, it trains a multi-scale spatial base evaluator using a target view that includes the original scale view and at least one controlled resampled view, enabling the base quality evaluation to simultaneously utilize the distortion-sensitive response at different observation scales. Then, it freezes the multi-scale spatial base evaluator and uses a spatiotemporal feature extraction branch and a residual prediction head to fuse the frame-by-frame spatial latent features and frame-by-frame spatiotemporal latent features corresponding to the temporal positions and estimate the frame-by-frame residual correction. The quality score is obtained by additively fusing the temporal aggregation results of the base quality response and the temporal aggregation results of the residual correction. This approach not only supplements cross-frame variation information but also maintains the existing spatial quality scale, reducing the optimization instability caused by the competition between the base quality evaluation and the temporal residual correction, thereby improving the accuracy and stability of HDR video quality evaluation.

[0056] In one specific implementation, the evaluator training module 11 includes: The connection feature extraction unit is used to extract features from the target view corresponding to each video frame of the training video data using the target feature extraction network of the initial evaluator to obtain the corresponding quality-aware vector, and to connect the quality-aware vectors corresponding to each target view of the video frame to obtain the target connection feature corresponding to the video frame. The first frame-by-frame analysis unit is used to determine the first frame-by-frame basic quality response corresponding to the training video data based on the target connectivity features corresponding to each video frame using the multi-scale feature fusion layer and basic quality mapping layer of the initial evaluator.

[0057] In one specific implementation, the evaluator training module 11 includes: The evaluator update unit is used to determine the first quality prediction loss corresponding to the first training set based on the first frame-by-frame basic quality response and the first subjective quality score corresponding to the training video data, and update the initial evaluator based on the first quality prediction loss to obtain the updated initial evaluator. The evaluator determination unit is used to obtain the evaluation result of the first correlation index corresponding to the first preset HDR verification set using the updated initial evaluator, and to determine the updated initial evaluator as the target evaluator when the evaluation result of the first correlation index meets the first iteration termination condition.

[0058] In one specific implementation, the training feature extraction module 12 includes: The second frame-by-frame analysis unit is used to obtain the second frame-by-frame basic quality response and frame-by-frame spatial latent features corresponding to each HDR continuous frame segment in the second training set using the target evaluator in the initial video quality evaluation model. The third frame-by-frame analysis unit is used to obtain the frame-by-frame spatiotemporal latent features corresponding to each of the HDR consecutive frame segments in the second training set by utilizing the spatiotemporal feature extraction branch of the initial video quality evaluation model.

[0059] In one specific implementation, the video quality evaluation module 14 includes: An aggregation unit is used to perform temporal aggregation on the second frame-by-frame basic quality response to obtain a first aggregation result, and to perform temporal aggregation on the frame-by-frame residual correction amount to obtain a second aggregation result; The training quality score determination unit is used to determine the training quality score corresponding to the HDR consecutive frame segment based on the first aggregation result and the second aggregation result.

[0060] In one specific implementation, the video quality evaluation module 14 includes: The prediction loss determination unit is used to determine target relevance constraints and target ranking constraints based on the training quality score and the second subjective quality score corresponding to each HDR consecutive frame segment in the second training set, and to determine the second quality prediction loss corresponding to the second training set based on the target relevance constraints and the target ranking constraints. The model update unit is used to update the spatiotemporal feature extraction branch and the residual prediction head of the initial video quality assessment model based on the second quality prediction loss to obtain the updated initial video quality assessment model. The model determination unit is used to obtain the evaluation result of the second correlation index corresponding to the second preset HDR validation set using the updated initial video quality evaluation model, and to determine the updated initial video quality evaluation model as the target video quality evaluation model when the evaluation result of the second correlation index meets the second iteration termination condition.

[0061] In one specific implementation, the video quality evaluation module 14 includes: The video segmentation unit is used to acquire the HDR video to be evaluated, and to decode, normalize the encoded value, normalize the spatial size, and perform deterministic temporal sampling on the HDR video to be evaluated to obtain the continuous frame segments to be evaluated corresponding to the HDR video to be evaluated; the continuous frame segments to be evaluated include at least one continuous frame segment of the HDR video to be evaluated. The video quality score determination unit is used to input the continuous frame segment to be evaluated into the target video quality evaluation model to obtain the target frame-by-frame basic quality response and the target frame-by-frame residual correction amount corresponding to the continuous frame segment to be evaluated, and to determine the segment-level quality score corresponding to the continuous frame segment to be evaluated based on the target frame-by-frame basic quality response and the target frame-by-frame residual correction amount. The evaluation result acquisition unit is used to determine the target video quality evaluation result corresponding to the HDR video to be evaluated based on the segment-level quality score corresponding to the continuous frame segment to be evaluated of the HDR video to be evaluated.

[0062] Furthermore, embodiments of this application also disclose an electronic device, Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0063] Figure 6This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the HDR video quality evaluation method based on residual timing correction disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be a computer.

[0064] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0065] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored thereon can include an operating system 221, computer programs 222, etc., and the storage method can be temporary storage or permanent storage.

[0066] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the HDR video quality evaluation method based on residual timing correction disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0067] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed HDR video quality evaluation method based on residual timing correction. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0068] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0069] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0070] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0071] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0072] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for evaluating the quality of HDR video based on residual temporal correction, characterized in that, include: Acquire training video data, construct a first training set based on the training video data, and construct a target view corresponding to each video frame in the first training set. Use an initial evaluator to obtain a first frame-by-frame basic quality response corresponding to the training video data based on the target view, and update the initial evaluator based on the first frame-by-frame basic quality response and the first subjective quality score corresponding to the training video data to obtain a target evaluator. The target view includes the original scale view corresponding to the video frame and at least one controlled resampling view. The initial evaluator is a pre-constructed multi-scale spatial basic evaluator. HDR consecutive frame segments are acquired, and a second training set is constructed using the HDR consecutive frame segments. The target evaluator is loaded into the initial video quality evaluation model, and the second frame-by-frame basic quality response, frame-by-frame spatial latent feature, and frame-by-frame spatiotemporal latent feature corresponding to each HDR consecutive frame segment in the second training set are obtained using the initial video quality evaluation model. The HDR consecutive frame segments carry a corresponding second subjective quality score. Using the residual prediction head of the initial video quality evaluation model, the frame-by-frame spatial latent features and the frame-by-frame spatiotemporal latent features corresponding to each other in the temporal position of the HDR continuous frame segment are fused to obtain the target joint features, and the frame-by-frame residual correction amount corresponding to the HDR continuous frame segment is determined based on the target joint features. The training quality score corresponding to the HDR continuous frame segment is obtained based on the second frame-by-frame basic quality response and the frame-by-frame residual correction. The initial video quality evaluation model is updated based on the training quality score and the second subjective quality score to obtain the target video quality evaluation model. The target video quality evaluation model is then used to obtain the target video quality evaluation result corresponding to the HDR video to be evaluated.

2. The HDR video quality evaluation method based on residual temporal correction according to claim 1, characterized in that, The step of obtaining the first frame-by-frame basic quality response corresponding to the training video data based on the target view using the initial evaluator includes: The target feature extraction network of the initial evaluator is used to extract features from the target view corresponding to each video frame of the training video data to obtain the corresponding quality-aware vector, and the quality-aware vectors corresponding to each target view of the video frame are connected to obtain the target connection features corresponding to the video frame. The first frame-by-frame basic quality response corresponding to the training video data is determined based on the target connectivity features corresponding to each video frame using the multi-scale feature fusion layer and basic quality mapping layer of the initial evaluator.

3. The HDR video quality evaluation method based on residual temporal correction according to claim 1, characterized in that, The step of updating the initial evaluator based on the first frame-by-frame basic quality response and the first subjective quality score corresponding to the training video data to obtain the target evaluator includes: Based on the first frame-by-frame basic quality response and the first subjective quality score corresponding to the training video data, the first quality prediction loss corresponding to the first training set is determined, and the initial evaluator is updated based on the first quality prediction loss to obtain the updated initial evaluator. The updated initial evaluator is used to obtain the evaluation result of the first correlation index corresponding to the first preset HDR validation set, and when the evaluation result of the first correlation index meets the first iteration termination condition, the updated initial evaluator is determined as the target evaluator.

4. The HDR video quality evaluation method based on residual temporal correction according to claim 1, characterized in that, The step of obtaining the second frame-by-frame basic quality response, frame-by-frame spatial latent features, and frame-by-frame spatiotemporal latent features corresponding to each HDR consecutive frame segment in the second training set using the initial video quality evaluation model includes: The target evaluator in the initial video quality evaluation model is used to obtain the second frame-by-frame basic quality response and frame-by-frame spatial latent features corresponding to each HDR continuous frame segment in the second training set. The spatiotemporal feature extraction branch of the initial video quality evaluation model is used to obtain the frame-by-frame spatiotemporal latent features corresponding to each HDR continuous frame segment in the second training set.

5. The HDR video quality evaluation method based on residual temporal correction according to claim 1, characterized in that, The step of obtaining the training quality score corresponding to the HDR consecutive frame segment based on the second frame-by-frame basic quality response and the frame-by-frame residual correction includes: The second frame-by-frame basic quality response is aggregated over time to obtain a first aggregation result, and the frame-by-frame residual correction is aggregated over time to obtain a second aggregation result; The training quality score corresponding to the HDR consecutive frame segment is determined based on the first aggregation result and the second aggregation result.

6. The HDR video quality evaluation method based on residual temporal correction according to claim 1, characterized in that, The step of updating the initial video quality assessment model based on the training quality score and the second subjective quality score to obtain the target video quality assessment model includes: Based on the training quality score and the second subjective quality score corresponding to each HDR consecutive frame segment in the second training set, a target relevance constraint and a target ranking constraint are determined, and a second quality prediction loss corresponding to the second training set is determined based on the target relevance constraint and the target ranking constraint. Based on the second quality prediction loss, the spatiotemporal feature extraction branch of the initial video quality assessment model and the residual prediction head are updated to obtain the updated initial video quality assessment model. The updated initial video quality evaluation model is used to obtain the evaluation result of the second correlation index corresponding to the second preset HDR validation set. When the evaluation result of the second correlation index meets the second iteration termination condition, the updated initial video quality evaluation model is determined as the target video quality evaluation model.

7. The HDR video quality evaluation method based on residual temporal correction according to any one of claims 1 to 6, characterized in that, The step of obtaining the target video quality evaluation result corresponding to the HDR video to be evaluated using the target video quality evaluation model includes: The HDR video to be evaluated is acquired, and the HDR video to be evaluated is decoded, the encoded value is normalized, the spatial size is regularized, and deterministic temporal sampling is performed to obtain the continuous frame segments to be evaluated corresponding to the HDR video to be evaluated; the continuous frame segments to be evaluated include at least one continuous frame segment of the HDR video to be evaluated. The continuous frame segments to be evaluated are input into the target video quality evaluation model to obtain the target frame-by-frame basic quality response and the target frame-by-frame residual correction amount corresponding to the continuous frame segments to be evaluated, and the segment-level quality score corresponding to the continuous frame segments to be evaluated is determined based on the target frame-by-frame basic quality response and the target frame-by-frame residual correction amount. The target video quality evaluation result is determined based on the segment-level quality score corresponding to the consecutive frame segments of the HDR video to be evaluated.

8. An HDR video quality evaluation device based on residual temporal correction, characterized in that, include: An evaluator training module is used to acquire training video data, construct a first training set based on the training video data, and build a target view corresponding to each video frame in the first training set. An initial evaluator is used to acquire a first frame-by-frame basic quality response corresponding to the training video data based on the target view. The initial evaluator is then updated based on the first frame-by-frame basic quality response and a first subjective quality score corresponding to the training video data to obtain a target evaluator. The target view includes the original scale view corresponding to the video frame and at least one controlled resampling view. The initial evaluator is a pre-constructed multi-scale spatial basic evaluator. The training feature extraction module is used to acquire HDR continuous frame segments, construct a second training set using the HDR continuous frame segments, load the target evaluator into the initial video quality evaluation model, and use the initial video quality evaluation model to acquire the second frame-by-frame basic quality response, frame-by-frame spatial latent features, and frame-by-frame spatiotemporal latent features corresponding to each HDR continuous frame segment in the second training set; the HDR continuous frame segments carry corresponding second subjective quality scores. The residual correction amount determination module is used to use the residual prediction head of the initial video quality evaluation model to fuse the frame-by-frame spatial latent features and the frame-by-frame spatiotemporal latent features that correspond to each other in the temporal position of the HDR continuous frame segment to obtain the target joint features, and to determine the frame-by-frame residual correction amount corresponding to the HDR continuous frame segment based on the target joint features. The video quality evaluation module is used to obtain the training quality score corresponding to the HDR continuous frame segment based on the second frame-by-frame basic quality response and the frame-by-frame residual correction amount, update the initial video quality evaluation model based on the training quality score and the second subjective quality score to obtain the target video quality evaluation model, and use the target video quality evaluation model to obtain the target video quality evaluation result corresponding to the HDR video to be evaluated.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the HDR video quality evaluation method based on residual timing correction as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the HDR video quality evaluation method based on residual timing correction as described in any one of claims 1 to 7.

Images