A video quality determination method, apparatus, device and readable storage medium

Abstract

The application discloses a video quality determination method and device, equipment and a readable storage medium. The application migrates an HDR domain to an SDR domain, combines training optimization, uses a model obtained by training an SDR video as a backbone network, combines a domain migration module to construct a model capable of performing quality evaluation on an HDR video, can reduce the complexity of model development and maintenance, can fully utilize pre-training knowledge of a large-scale standard dynamic range dataset, and reduces the dependence on high dynamic range data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a video quality determination method and device, equipment and readable storage medium. BACKGROUND

[0002] Full-reference video quality assessment refers to a method of learning the quality of a video by comparing the differences between an original reference video and a distorted video. That is, the original reference video and the distorted video are input, and a quality score is obtained as output, and the obtained result needs to be as consistent as possible with the subjective perception of the human eye.

[0003] However, the current high dynamic range (HDR) / standard dynamic range (SDR) cross-domain adaptability is insufficient. For example, a large number of full-reference quality evaluation algorithms are designed for standard dynamic range videos, while high dynamic range videos have significant differences in pixel distribution, brightness dynamic range, and color space from standard dynamic range, which makes it difficult to directly migrate standard dynamic range models, resulting in problems such as slow response to details and distorted scores. Designing different models for high dynamic range and standard dynamic range videos respectively not only increases the complexity of model development and maintenance.

[0004] In addition, the scarcity of high dynamic range data makes it difficult to train, that is, the high dynamic range video quality evaluation dataset is relatively scarce, and the amount of data is limited. Directly training a model on a small amount of high dynamic range data can easily lead to overfitting problems, making it difficult to obtain good generalization performance. At the same time, the annotation cost of high dynamic range video is high, and it is difficult to obtain large-scale labeled data, which further limits the improvement of model performance.

[0005] In summary, how to effectively solve the problem of HDR video quality evaluation is a technical problem that needs to be solved by the technical personnel in the field. SUMMARY

[0006] The purpose of the present application is to provide a video quality determination method, device, equipment and readable storage medium, which can reduce the complexity of model development and maintenance by using domain migration and training optimization, using the model trained by SDR video as the backbone network, and combining the domain migration module to construct a model that can evaluate the quality of HDR video. It can not only reduce the complexity of model development and maintenance, but also make full use of the pre-training knowledge of large-scale standard dynamic range dataset, and reduce the dependence on high dynamic range data.

[0007] To solve the above technical problems, the present application provides the following technical solutions:

[0008] A video quality evaluation method comprises:

[0009] acquire a target video, and determine a dynamic range of the target video; wherein the dynamic range comprises an HDR and an SDR, the HDR being a high dynamic range, and the SDR being a standard dynamic range;

[0010] in a case where the dynamic range of the target video is the HDR, performing mapping processing on the target video by using a domain migration module in a trained video quality evaluation model, performing evaluation processing on a video obtained by the mapping by using a backbone model of the video quality evaluation model, and obtaining a quality evaluation result;

[0011] wherein a process of training the video quality evaluation model comprises:

[0012] performing quality evaluation training on the backbone network by using an SDR video training set;

[0013] after the training of the backbone network is completed, adding the domain migration module on the basis of the backbone network;

[0014] in a process of performing quality evaluation training on the video quality evaluation model by using a mixed video set comprising SDR videos and HDR videos, if a current video is an HDR video, performing mapping processing on the current HDR video by using the domain migration module, and processing a video obtained by the mapping by using the backbone network;

[0015] performing fine-tuning on the video quality evaluation model by using an HDR video training set, and obtaining the video quality evaluation model.

[0016] Preferably, performing fine-tuning on the video quality evaluation model by using an HDR video training set, and obtaining the video quality evaluation model, comprises:

[0017] freezing model parameters of the backbone network, performing fine-tuning on a domain migration module in the video quality evaluation model by using the HDR video training set, and obtaining the video quality evaluation model;

[0018] Correspondingly, it also comprises:

[0019] in a case where the dynamic range of the target video is the SDR, performing evaluation processing on the target video by using a backbone model in a trained video quality evaluation model, and obtaining a quality evaluation result.

[0020] Preferably, performing fine-tuning on the video quality evaluation model by using an HDR video training set, and obtaining the video quality evaluation model, comprises:

[0021] performing fine-tuning on the domain migration module and the backbone network by using the HDR video training set, and obtaining the video quality evaluation model.

[0022] Preferably, the domain migration module comprises a plurality of residual blocks, and each residual block comprises a convolution layer, an activation function and a residual connection.

[0023] Preferably, the video quality evaluation model is obtained by fine-tuning a video quality evaluation model using an HDR video training set, and the fine-tuning comprises:

[0024] The training sample and the verification sample are obtained using the HDR video training set.

[0025] The video quality evaluation model is fine-tuned using the training sample.

[0026] The video quality evaluation model is verified using the verification sample.

[0027] After verification, the current video quality evaluation model is determined as the video quality evaluation model.

[0028] Preferably, the training sample and the verification sample are obtained using the SDR video training set, and the obtaining comprises:

[0029] The SDR videos in the SDR video training set are randomly frame-extracted to obtain the training sample.

[0030] The SDR videos in the SDR video training set are fixedly frame-extracted to obtain the verification sample.

[0031] Preferably, after the target video is processed using the domain migration module in the trained video quality evaluation model, the video obtained by the processing is evaluated using the backbone model of the video quality evaluation model to obtain a quality evaluation result, and the evaluating comprises:

[0032] The reference video and the distorted video corresponding to the target video are frame-by-frame extracted to obtain a current frame.

[0033] The high dynamic range feature of the current frame is mapped to a standard dynamic range domain using the domain migration module.

[0034] The image feature mapped to the standard dynamic range domain is feature-extracted, feature-fused and quality-regressed using the backbone network to obtain a single-frame evaluation result of the current frame.

[0035] The single-frame evaluation results corresponding to each frame of the target video are aggregated to obtain the quality evaluation result.

[0036] A video quality determination apparatus comprises:

[0037] The video acquisition module is configured to acquire a target video and determine a dynamic range of the target video, wherein the dynamic range includes an HDR (high dynamic range) and an SDR (standard dynamic range).

[0038] The HDR evaluation module is configured to, when the dynamic range of the target video is the HDR, perform mapping processing on the target video by using a domain migration module in a trained video quality evaluation model, perform evaluation processing on the mapped video by using a backbone model of the video quality evaluation model, and obtain a quality evaluation result.

[0039] The model training module includes a first-stage training unit, a second-stage training unit and a third-stage training unit.

[0040] The first-stage training unit is configured to perform quality evaluation training on the backbone network by using an SDR video training set.

[0041] The second-stage training unit is configured to, after the training of the backbone network is completed, add the domain migration module to the backbone network, and in the process of performing quality evaluation training on the video quality evaluation model by using a mixed video set including SDR videos and HDR videos, perform mapping processing on a current HDR video by using the domain migration module and perform processing on the mapped video by using the backbone network, if the current video is the HDR video.

[0042] The third-stage training unit is configured to fine-tune the video quality evaluation model by using an HDR video training set, and obtain the video quality evaluation model.

[0043] An electronic device includes:

[0044] A memory configured to store a computer program.

[0045] A processor configured to implement the steps of the video quality determination method when the computer program is executed.

[0046] A readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the video quality determination method.

[0047] The method provided in the embodiment of the application is applied to obtain a target video and determine a dynamic range of the target video; wherein the dynamic range includes HDR and SDR, the HDR is high dynamic range, and the SDR is standard dynamic range; in the case that the dynamic range of the target video is HDR, a domain migration module in a trained video quality evaluation model is used to perform mapping processing on the target video, and then a backbone model of the video quality evaluation model is used to perform evaluation processing on the video obtained through the mapping, to obtain a quality evaluation result; wherein the process of training the video quality evaluation model includes: using an SDR video training set to perform quality evaluation training on the backbone network; after the training of the backbone network is completed, a domain migration module is added on the basis of the backbone network; in the process of using a mixed video set including SDR videos and HDR videos to perform quality evaluation training on the video quality evaluation model, if a current video is an HDR video, the domain migration module is used to perform mapping processing on the current HDR video, and the backbone network is used to process the video obtained through the mapping; the video quality evaluation model is fine-tuned using an HDR video training set, to obtain the video quality evaluation model.

[0048] In the application, a video quality evaluation model can be trained in advance, and the backbone network of the model can be obtained by performing quality evaluation training on an SDR video training set. Then, in order to enable the backbone network to process DHR videos, a domain migration module can be set, which maps the HDR videos in a mixed video set including SDR videos and HDR videos into SDR videos and then processes them. Finally, the video quality evaluation model is fine-tuned based on an HDR video training set, to obtain the video quality evaluation model. After the target video is obtained, its dynamic range can be determined. That is, it is determined whether the target video corresponds to HDR or SDR. In the case that it is clear that the target video corresponds to HDR, the domain migration module in the trained video quality evaluation model is used to perform mapping processing on the target video, and then the backbone model of the video quality evaluation model is used to perform evaluation processing on the video obtained through the mapping, to obtain a quality evaluation result.

[0049] The model obtained by training and optimization through domain migration and using the model trained by SDR videos as the backbone network, in combination with the domain migration module, can not only reduce the complexity of model development and maintenance, but also fully utilize the pre-training knowledge of large-scale standard dynamic range data sets and reduce the dependence on high dynamic range data.

[0050] Correspondingly, the embodiments of the application also provide a video quality determination apparatus, device and readable storage medium corresponding to the video quality determination method, which have the above technical effects and will not be described herein. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to make the technical scheme of the embodiments of the present application or the related art clearer, the accompanying drawings needed in the embodiments or the related art description will be briefly introduced. Obviously, the accompanying drawings in the description are only some embodiments of the present application, and all other embodiments obtained by those of ordinary skill in the art without creative work based on the accompanying drawings belong to the protection scope of the present application.

[0052] Figure 1 An implementation flowchart of a video quality determination method in an embodiment of the present application is shown in FIG. 1.

[0053] Figure 2 An implementation flowchart of a video quality determination method in an embodiment of the present application is shown in FIG. 1.

[0054] Figure 3 An implementation flowchart of a video quality determination method in an embodiment of the present application is shown in FIG. 1.

[0055] Figure 4 An implementation flowchart of a video quality determination method in an embodiment of the present application is shown in FIG. 1.

[0056] Figure 5 An implementation flowchart of a video quality determination method in an embodiment of the present application is shown in FIG. 1.

[0057] Figure 6 An implementation flowchart of a video quality determination method in an embodiment of the present application is shown in FIG. 1.

[0058] Figure 7 An implementation flowchart of a video quality determination method in an embodiment of the present application is shown in FIG. 1. DETAILED DESCRIPTION

[0059] In order to make the technical scheme of the embodiments of the present application or the related art clearer, the accompanying drawings needed in the embodiments or the related art description will be briefly introduced. Obviously, the accompanying drawings in the description are only some embodiments of the present application, and all other embodiments obtained by those of ordinary skill in the art without creative work based on the accompanying drawings belong to the protection scope of the present application.

[0060] Please refer to Figure 1 , Figure 1 An implementation flowchart of a video quality determination method in an embodiment of the present application is shown in FIG. 1.

[0061] S101, a target video is acquired, and a dynamic range of the target video is determined.

[0062] The dynamic range includes HDR and SDR, where HDR is high dynamic range, and SDR is standard dynamic range.

[0063] It should be noted that in the embodiment, the video quality is evaluated by using the full reference video quality method, and therefore the target video actually includes the reference video and the distorted video.

[0064] In the case where the video quality needs to be evaluated, the target video to be evaluated is obtained, and the dynamic range of the target video is determined, i.e., the dynamic range of the reference video is determined. The dynamic ranges of the reference video and the distorted video are consistent.

[0065] S102, in the case where the dynamic range of the target video is HDR, the domain migration module in the trained video quality evaluation model is used to perform mapping processing on the target video, and then the backbone model of the video quality evaluation model is used to perform evaluation processing on the mapped video, to obtain a quality evaluation result.

[0066] The mapped video is a video with SDR domain similar features, and the video can be regarded as an SDR feature video. That is, the subsequent backbone network can process the mapped video as an SDR video.

[0067] In the case where the dynamic range of the target video corresponds to HDR, when only the video quality is analyzed, the domain migration module in the trained video quality evaluation model is used to perform mapping processing on the target video, and then the backbone model of the video quality evaluation model is used to perform evaluation processing on the mapped video, to obtain a quality evaluation result.

[0068] Please refer to Figure 2 The process of training the video quality evaluation model includes.

[0069] In the first stage, the backbone network is trained by using the SDR video training set.

[0070] In the second stage, after the training of the backbone network is completed, the domain migration module is added on the basis of the backbone network.

[0071] In the process of training the video quality evaluation model by using the mixed video set including the SDR video and the HDR video, if the current video is an HDR video, the domain migration module is used to perform mapping processing on the current HDR video, and the backbone network is used to process the mapped video.

[0072] In the third stage, the video quality evaluation model is fine-tuned by using the HDR video training set, to obtain the video quality evaluation model.

[0073] The specific implementation of different training stages will be described in detail below.

[0074] In the first stage, the backbone network is pre-trained.

[0075] In the first stage of training, the main goal is to pre-train the backbone network (shown as the main quality evaluation model) using a large-scale standard dynamic range quality evaluation dataset, establishing a basic quality evaluation capability. Figure 2

[0076] Training goal: Train the backbone network (including feature extraction and fusion modules) to enable it to extract effective quality perception features from standard dynamic range videos.

[0077] Training data: Use a large-scale standard dynamic range video quality evaluation dataset, which contains a large number of standard dynamic range video samples and their corresponding quality scores, providing rich training samples for the model.

[0078] Training method: Train all parameters of the backbone network end-to-end, minimize the mean square error (MSE Loss) between the predicted quality score and the true quality score, and make the model learn the quality evaluation rules of standard dynamic range videos.

[0079] Output result: After training, save the pre-training weights of the backbone network as the initialization parameters for subsequent stage training, providing a good feature extraction basis for the model.

[0080] Second stage: Standard dynamic range and high dynamic range joint training.

[0081] In the second stage of training, high dynamic range data is introduced, and standard dynamic range data and high dynamic range data are used for joint training, enabling the model to handle both types of videos.

[0082] Pre-training weight loading: Load the pre-training weights of the backbone network obtained in the first stage of training as the model initialization, using the learned standard dynamic range quality evaluation knowledge.

[0083] Domain migration module introduction: Introduce a domain migration module to map high dynamic range image features to the standard dynamic range domain, enabling high dynamic range data to be uniformly processed with standard dynamic range data in the same backbone network.

[0084] Mixed data training: Use a video dataset containing standard dynamic range and high dynamic range mixed, through joint training, make the model learn the quality evaluation rules of two types of data at the same time.

[0085] Training strategy: Domain migration module and backbone network are trained at the same time, through end-to-end backpropagation, optimize the quality evaluation ability of the whole model for mixed data.

[0086] ​Training objective: Minimize the mean squared error (MSE Loss) between the predicted quality score and the true quality score, enabling the model to accurately predict the quality of standard dynamic range and high dynamic range videos.

[0087] Third stage: High dynamic range data fine-tuning.

[0088] In the third stage of training, the model is fine-tuned for high dynamic range data to further improve its performance on high dynamic range video quality evaluation tasks.

[0089] Pre-training weight loading: Load the optimal model weights obtained from the second stage of training, including the parameters of the domain transfer module and the backbone network, as the initialization for fine-tuning.

[0090] Training data: Use high dynamic range video quality evaluation datasets, focusing on the quality evaluation task of high dynamic range data, to make the model better adapt to the characteristics of high dynamic range videos.

[0091] Fine-tuning strategy: Fine-tune the model with a smaller learning rate to make fine adjustments to the high dynamic range data based on the existing knowledge.

[0092] Training objective: Further optimize the model's performance on high dynamic range video quality evaluation tasks and improve its ability to perceive high dynamic range video quality features.

[0093] Through this three-stage progressive training strategy, the model can start from the basic quality evaluation ability of standard dynamic range data, gradually expand to mixed data, and finally make fine adjustments to high dynamic range data, realizing a gradual learning process from general to specific. The first stage fully utilizes large-scale standard dynamic range datasets to establish a basic quality evaluation ability; the second stage realizes unified processing of two types of data through the domain transfer module, enabling the model to learn the quality evaluation rules of both types of data; the third stage fine-tunes specifically for high dynamic range data to further improve the model's performance on high dynamic range tasks. In addition, the random frame extraction strategy improves data utilization efficiency: randomly extract N frames from each video during training and fix M frames during validation, which improves data utilization efficiency and ensures evaluation stability. This phased training strategy avoids the overfitting problem that may occur when directly training on small-scale high dynamic range data, and improves the stability and final performance of the training through a gradual learning process from general to specific.

[0094] That is, in the embodiment of the present application, a model capable of quality evaluation of HDR video can be pre-trained. In order to reduce the difficulty of model development and reduce the dependence on HDR samples, a model capable of quality evaluation of SDR video can be directly used as a backbone network, that is, the backbone network can be trained for quality evaluation using an SDR video training set.

[0095] After the backbone network is trained, a domain migration module can be added based on the backbone network. Then, the video quality evaluation model is trained for quality evaluation using a mixed video set. For SDR video, the training process is the same as the first stage, and for HDR video, the domain migration module is used to map the current HDR video, and the backbone network is used to process the mapped video.

[0096] Finally, the video quality evaluation model is fine-tuned by the HDR video training set to obtain the video quality evaluation model. In this way, the video quality evaluation model can evaluate the quality of the HDR video.

[0097] The domain migration module includes a plurality of residual blocks, and each residual block includes a convolution layer, an activation function, and a residual connection. The domain migration module mainly solves the domain difference problem between high dynamic range video and standard dynamic range video. The module converts the feature representation of the high dynamic range image into a feature representation similar to that of the standard dynamic range image through a lightweight convolutional network structure.

[0098] Please refer to Figure 3 The domain migration module adopts a lightweight design of a basic residual block and is stacked by a plurality of basic residual blocks. Each basic residual block includes a convolution layer, an activation function, and a residual connection, and maps the high dynamic range image features to the standard dynamic range domain through residual learning. The input and output dimensions of the module are consistent, both being 3xHxW, which ensures the compatibility with the backbone quality evaluation model.

[0099] The working principle of the domain migration module is to learn the mapping relationship of the high dynamic range image to the standard dynamic range domain through the stacking of a plurality of basic residual blocks. Each basic residual block includes a convolution layer and an activation function, and adds the original input and the convolution processed result through a residual connection to realize residual learning. This design enables the module to learn the difference between high dynamic range and standard dynamic range, and realizes domain migration by adding the original input and the learned difference. The input and output dimensions of the module are consistent (3xHxW), which ensures the compatibility with the backbone quality evaluation model. This lightweight design not only ensures the domain migration effect, but also keeps the parameter quantity and computational overhead of the model within an acceptable range.

[0100] The method provided in the embodiment of the application is applied to obtain a target video and determine a dynamic range of the target video; the dynamic range includes HDR and SDR, the HDR is a high dynamic range, and the SDR is a standard dynamic range; in a case where the dynamic range of the target video is the HDR, a domain migration module in a trained video quality evaluation model is used to perform mapping processing on the target video, a backbone model of the video quality evaluation model is used to perform evaluation processing on a video obtained by the mapping, and a quality evaluation result is obtained; a process of training the video quality evaluation model includes: using an SDR video training set to perform quality evaluation training on the backbone network; after the training of the backbone network is completed, a domain migration module is added on the basis of the backbone network; in a process of using a mixed video set including SDR videos and HDR videos to perform quality evaluation training on the video quality evaluation model, if a current video is an HDR video, the domain migration module is used to perform mapping processing on the current HDR video, and the backbone network is used to process a video obtained by the mapping; the video quality evaluation model is fine-tuned using an HDR video training set, and the video quality evaluation model is obtained.

[0101] In the application, a video quality evaluation model can be trained in advance, a backbone network of the model can be obtained by performing quality evaluation training on an SDR video training set. Then, in order to enable the backbone network to process DHR videos, a domain migration module can be set, and the HDR videos in a mixed video set including SDR videos and HDR videos are mapped to SDR videos, and then processed. Finally, the video quality evaluation model is fine-tuned based on an HDR video training set, and the video quality evaluation model is obtained. After the target video is obtained, the dynamic range thereof can be determined. That is, it is determined whether the target video corresponds to the HDR or the SDR. In a case where it is clear that the target video corresponds to the HDR, the domain migration module in the trained video quality evaluation model is used to perform mapping processing on the target video, and then the backbone model of the video quality evaluation model is used to perform evaluation processing on a video obtained by the mapping, and a quality evaluation result is obtained.

[0102] The model obtained by training and optimization through domain migration and using the model obtained by training on SDR videos as a backbone network and combining a domain migration module to construct a model capable of performing quality evaluation on HDR videos not only can reduce the complexity of model development and maintenance, but also can fully utilize the pre-training knowledge of a large-scale standard dynamic range data set and reduce the dependence on high dynamic range data.

[0103] It should be noted that based on the above embodiments, the application embodiments also provide corresponding improvement schemes. The steps involved in the preferred / modified embodiments can be mutually referred to, and the corresponding beneficial effects can also be mutually referred to. The preferred / modified embodiments are not described one by one herein.

[0104] In an embodiment of the present application, after the target video is mapped by using the domain migration module in the trained video quality evaluation model, the video obtained by mapping is evaluated by using the backbone model of the video quality evaluation model to obtain a quality evaluation result, including: extracting the reference video and the distorted video corresponding to the target video frame by frame to obtain a current frame; mapping the high dynamic range feature of the current frame to the standard dynamic range domain by using the domain migration module; performing feature extraction, feature fusion and quality regression processing on the image feature mapped to the standard dynamic range domain by using the backbone network to obtain a single-frame evaluation result of the current frame; and aggregating the single-frame evaluation results corresponding to each frame of the target video to obtain the quality evaluation result.

[0105] Please refer to Figure 4 , input a reference video sequence (i.e. a reference video) and a distorted video sequence (i.e. a distorted video), and the two video sequences are preprocessed by a video preprocessing link, and then the picture extraction operation is performed frame by frame to convert the video sequence into an image frame sequence. For each extracted frame image, first determine whether the frame is a high dynamic range. If the result of the determination is yes, the frame image enters the domain migration module for processing to map the high dynamic range feature to the standard dynamic range domain; if the result of the determination is no, the domain migration module is directly skipped. The image frame processed by the domain migration or directly skipped then enters a unified backbone quality evaluation model for feature extraction and quality evaluation. The backbone quality evaluation model extracts multi-scale features and performs fusion processing, and calculates a single-frame quality score (i.e. a specific single-frame evaluation result) through a quality regression network. Finally, the quality scores of all frames are aggregated for processing, the mean value of all frame scores is calculated, and the overall quality score of the video (a specific quality evaluation result) is obtained as the final output.

[0106] In an embodiment of the present application, the video quality evaluation model is fine-tuned by using an HDR video training set, including:

[0107] The model parameters of the backbone network are frozen, and the domain migration module in the video quality evaluation model is fine-tuned by using the HDR video training set to obtain the video quality evaluation model;

[0108] Correspondingly, it also includes:

[0109] In the case where the dynamic range of the target video is SDR, the target video is evaluated by using the backbone model in the trained video quality evaluation model to obtain a quality evaluation result.

[0110] In actual application, the video quality evaluation model can also be used to evaluate the quality of the SDR video. In order to balance the quality evaluation reliability of the SDR video and the HDR video, when fine-tuning the video quality evaluation model, only the domain migration module can be fine-tuned, that is, by freezing the model parameters of the backbone network, the video quality evaluation model can keep the reliability of the quality evaluation of the SDR video; by fine-tuning the domain migration module, the reliability of the quality evaluation of the model when processing the HDR video can be ensured.

[0111] In addition, during the quality evaluation process, the evaluation processing of the SDR video will not be processed by the domain migration module, so fine-tuning the domain migration module will not affect the quality evaluation reliability of the SDR video.

[0112] In one specific embodiment of the present application, the video quality evaluation model is fine-tuned by using the HDR video training set, and the video quality evaluation model is obtained, including:

[0113] The domain migration module and the backbone network are fine-tuned by using the HDR video training set, and the video quality evaluation model is obtained.

[0114] That is, when the video quality evaluation requirement is mainly for quality evaluation of the HDR video, that is, the video quality evaluation model obtained by training does not need to evaluate the quality of the SDR video, at this time, the domain migration module and the backbone network can be fine-tuned by using the HDR video training set, so that the domain migration module and the backbone network are more suitable for the quality evaluation processing of the HDR video.

[0115] In one specific embodiment of the present application, the video quality evaluation model is fine-tuned by using the HDR video training set, and the video quality evaluation model is obtained, including:

[0116] The training sample and the verification sample are obtained by using the HDR video training set;

[0117] The video quality evaluation model is fine-tuned by using the training sample;

[0118] The video quality evaluation model is verified by using the verification sample;

[0119] After verification, the current video quality evaluation model is determined as the video quality evaluation model.

[0120] The training sample and the verification sample are obtained by using the SDR video training set, including:

[0121] The SDR video in the SDR video training set is randomly frame-sampled to obtain the training sample;

[0122] The SDR video in the SDR video training set is fixed frame-sampled to obtain the verification sample.

[0123] That is, in the present embodiment, in order to improve the data utilization efficiency and the model generalization capability in the training process, a random frame extraction strategy for video data can be adopted. In the training phase, a random frame extraction method is used to select training samples for each video. Specifically, for the reference video (the video training set includes reference videos and distorted videos), N frames are randomly extracted from the video sequence as training samples. This random selection method can increase the diversity of training data and improve the generalization capability of the model. The random frame extraction strategy is essentially a data augmentation method. By selecting frames from different positions in the same video, the diversity of training samples is increased, which helps the model to learn more robust feature representations.

[0124] In the validation phase, a fixed frame extraction method is used to ensure consistency in evaluation. In the validation phase, M frames are fixedly sampled from each video for evaluation to ensure consistency and repeatability of the validation process.

[0125] Corresponding to the above method embodiment, the present embodiment also provides a video quality determination device. The video quality determination device described below can be mutually corresponding to the video quality determination method described above.

[0126] Referring to Figure 5 The device includes the following modules:

[0127] The video acquisition module 101 is configured to acquire a target video and determine the dynamic range of the target video. The dynamic range includes HDR and SDR. HDR refers to high dynamic range, and SDR refers to standard dynamic range.

[0128] The HDR evaluation module 102 is configured to, when the dynamic range of the target video is HDR, perform mapping processing on the target video by using a domain migration module in a trained video quality evaluation model, perform evaluation processing on the mapped video by using a backbone model of the video quality evaluation model, and obtain a quality evaluation result.

[0129] The model training module 103 includes a first stage training unit, a second stage training unit, and a third stage training unit.

[0130] The first stage training unit 1031 is configured to perform quality evaluation training on the backbone network by using an SDR video training set.

[0131] The second stage training unit 1032 is configured to, after completing the training of the backbone network, add a domain migration module to the backbone network. During the quality evaluation training of the video quality evaluation model by using a mixed video set including SDR videos and HDR videos, if the current video is an HDR video, the domain migration module is used to perform mapping processing on the current HDR video, and the backbone network is used to process the mapped video.

[0132] The third stage training unit 1033 is configured to fine-tune the video quality evaluation model by using the HDR video training set, to obtain the video quality evaluation model.

[0133] The device provided in the embodiments of the present application is used to obtain a target video and determine the dynamic range of the target video; the dynamic range includes HDR and SDR, where HDR is high dynamic range and SDR is standard dynamic range; in the case where the dynamic range of the target video is HDR, the domain migration module in the trained video quality evaluation model is used to perform mapping processing on the target video, and then the backbone model of the video quality evaluation model is used to perform evaluation processing on the mapped video to obtain a quality evaluation result; the process of training the video quality evaluation model includes: using the SDR video training set to perform quality evaluation training on the backbone network; after the training of the backbone network is completed, a domain migration module is added to the backbone network; during the quality evaluation training of the video quality evaluation model by using the mixed video set including SDR videos and HDR videos, if the current video is an HDR video, the domain migration module is used to perform mapping processing on the current HDR video, and the backbone network is used to process the mapped video; the video quality evaluation model is fine-tuned by using the HDR video training set, to obtain the video quality evaluation model.

[0134] In the present application, a video quality evaluation model can be pre-trained, and the backbone network of the model can be obtained by performing quality evaluation training based on an SDR video training set. Then, in order to enable the backbone network to process DHR videos, a domain migration module can be set to map the HDR videos in the mixed video set including SDR videos and HDR videos to SDR videos for processing. Finally, the video quality evaluation model is fine-tuned based on the SDR video training set, to obtain the video quality evaluation model. After the target video is obtained, the dynamic range thereof can be determined, i.e., whether the target video corresponds to HDR or SDR. In the case where it is clear that the target video corresponds to HDR, the domain migration module in the trained video quality evaluation model is used to perform mapping processing on the target video, and then the backbone model of the video quality evaluation model is used to perform evaluation processing on the mapped video to obtain a quality evaluation result.

[0135] By using domain migration and training optimization, the model obtained by training SDR videos is used as the backbone network, and a domain migration module is combined to construct a model capable of performing quality evaluation on HDR videos, which not only can reduce the complexity of model development and maintenance, but also can fully utilize the pre-training knowledge of large-scale standard dynamic range data sets and reduce the dependence on high dynamic range data.

[0136] In an embodiment of the present application, the third stage training unit is specifically configured to fine-tune the domain migration module in the video quality evaluation model by using the HDR video training set, and obtain the video quality evaluation model.

[0137] Correspondingly, the SDR evaluation module is further included, which is configured to perform evaluation processing on the target video by using the backbone model in the trained video quality evaluation model when the dynamic range of the target video is SDR, and obtain the quality evaluation result.

[0138] In an embodiment of the present application, the third stage training unit is specifically configured to fine-tune the domain migration module and the backbone network by using the HDR video training set, and obtain the video quality evaluation model.

[0139] In an embodiment of the present application, the domain migration module includes a plurality of residual blocks, and each residual block includes a convolution layer, an activation function and a residual connection.

[0140] In an embodiment of the present application, the third stage training unit is specifically configured to obtain training samples and verification samples by using the HDR video training set; fine-tune the video quality evaluation model by using the training samples; verify the video quality evaluation model by using the verification samples; and determine the current video quality evaluation model as the video quality evaluation model after verification.

[0141] In an embodiment of the present application, the third stage training unit is specifically configured to randomly frame the SDR videos in the SDR video training set to obtain training samples; and fixedly frame the SDR videos in the SDR video training set to obtain verification samples.

[0142] In an embodiment of the present application, the HDR evaluation module is specifically configured to frame by frame extract the reference video and the distorted video corresponding to the target video to obtain a current frame; map the high dynamic range features of the current frame to the standard dynamic range domain by using the domain migration module; perform feature extraction, feature fusion and quality regression processing on the image features mapped to the standard dynamic range domain by using the backbone network to obtain a single-frame evaluation result of the current frame; and aggregate the single-frame evaluation results corresponding to each frame of the target video to obtain the quality evaluation result.

[0143] It can be seen that the technical scheme provided by the embodiment of the application overcomes the insufficient adaptability of high dynamic range / standard dynamic range cross-domain, that is, a domain migration module is introduced in the model design link, and the module adopts a lightweight residual convolutional network structure to convert the feature representation of a high dynamic range image into a feature representation similar to that of a standard dynamic range image. Specifically, for an input video frame, the system first determines whether the frame is a high dynamic range. If the determination result is yes, the frame image enters the domain migration module for processing, and the high dynamic range feature is mapped to the standard dynamic range domain; if the determination result is no, the domain migration module is directly skipped. The image frame processed by the domain migration or directly skipped is then input into a unified backbone network for feature extraction and quality evaluation, so that the high dynamic range and standard dynamic range data can be uniformly processed in the same backbone network, avoiding the complexity of designing a model for different types of data, and improving the generality and deployment efficiency of the algorithm.

[0144] The technical scheme provided by the embodiment of the application also overcomes the difficulty in training caused by the scarcity of high dynamic range data, that is, a three-stage progressive training strategy is adopted. In the first stage, a large-scale standard dynamic range quality evaluation dataset is used to pre-train the backbone network to establish a basic quality evaluation capability. In the second stage, the domain migration module is introduced, and a mixed dataset of standard dynamic range and high dynamic range is used for joint training, so that the two types of data are uniformly processed through the domain migration module, and the model learns the quality evaluation rules of the two types of data. In the third stage, the high dynamic range data is specially fine-tuned, and a small learning rate is used for fine optimization, so that the existing knowledge is maintained, and the characteristics of the high dynamic range data are optimized.

[0145] In addition, the data utilization efficiency is improved by the random frame extraction strategy: N frames are randomly extracted from each video in the training stage, and M frames are fixedly sampled in the verification stage, which improves the data utilization efficiency and ensures the stability of the evaluation. This phased training strategy avoids the overfitting problem that may be caused by directly training on a small-scale high dynamic range data, and improves the stability and final performance of the training through a gradual learning process from general to specific. Wherein, M and N can be set according to actual training requirements, and are not specifically limited in the present application.

[0146] Corresponding to the above method embodiment, the embodiment of the application also provides an electronic device. The electronic device described below can be mutually corresponding with reference to the video quality determination method described above.

[0147] Referring to Figure 6 The electronic device includes:

[0148] The memory 332 is configured to store a computer program.

[0149] The processor 322 is configured to implement methods of determining video quality according to the embodiments described above when the computer programs are executed.

[0150] Specifically, refer to Figure 7 , Figure 7 A specific structural diagram of an electronic device is provided for the embodiment. The electronic device can have great differences due to different configurations or performances, and can include one or more processors (central processing units, CPU) (for example, one or more processors) and a memory 332 storing one or more computer programs 342 or data 344. The memory 332 can be temporary storage or persistent storage. The programs stored in the memory 332 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the data processing device. Further, the processor 322 can be configured to communicate with the memory 332 and execute a series of instruction operations in the memory 332 on the electronic device 301.

[0151] The electronic device 301 can also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.

[0152] The steps in the video quality determination method described above can be implemented by the structure of the electronic device.

[0153] Corresponding to the above method embodiments, the embodiments of the present application also provide a readable storage medium. The readable storage medium described below can be mutually corresponding with the video quality determination method described above.

[0154] A readable storage medium, the readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps of the video quality determination method according to the above method embodiments.

[0155] The readable storage medium can be a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various readable storage media that can store program codes.

[0156] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0157] Those skilled in the art can further understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in a general manner. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0158] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, a software module executed by a processor, or a combination of both. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0159] Finally, it should be noted that, in this document, relationships such as first and second are intended to distinguish one entity or operation from another, and do not necessarily require or imply any actual such relationship or order between the entities or operations. Moreover, the term including, comprising or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or apparatus including a series of elements includes not only those elements, but also other elements not explicitly listed, or other elements inherent in such a process, method, article or apparatus.

[0160] The principles and implementation manners of the present application are described by applying specific examples in this document, and the above description of the examples is only for the purpose of helping to understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in specific implementation manners and application ranges; in summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method of video quality determination, characterized by, The method comprises the following steps: acquiring a target video and determining a dynamic range of the target video; wherein the dynamic range comprises an HDR (High Dynamic Range) and an SDR (Standard Dynamic Range); in a case where the dynamic range of the target video is the HDR, performing mapping processing on the target video by using a domain migration module in a trained video quality evaluation model, performing evaluation processing on a video obtained by the mapping by using a backbone model of the video quality evaluation model, and obtaining a quality evaluation result; wherein a process of training the video quality evaluation model comprises: performing quality evaluation training on the backbone network by using an SDR video training set; after the training of the backbone network is completed, adding the domain migration module on the basis of the backbone network; in a process of performing quality evaluation training on the video quality evaluation model by using a mixed video set comprising SDR videos and HDR videos, if a current video is an HDR video, performing mapping processing on the current HDR video by using the domain migration module, and processing a video obtained by the mapping by using the backbone network; performing fine-tuning on the video quality evaluation model by using an HDR video training set, and obtaining the video quality evaluation model.

2. The method of claim 1, wherein, performing fine-tuning on the video quality evaluation model by using an HDR video training set, and obtaining the video quality evaluation model, comprises: freezing model parameters of the backbone network, performing fine-tuning on a domain migration module in the video quality evaluation model by using the HDR video training set, and obtaining the video quality evaluation model; correspondingly, the method further comprises: in a case where the dynamic range of the target video is the SDR, performing evaluation processing on the target video by using a backbone model in a trained video quality evaluation model, and obtaining a quality evaluation result.

3. The method of claim 1, wherein, performing fine-tuning on the video quality evaluation model by using an HDR video training set, and obtaining the video quality evaluation model, comprises: performing fine-tuning on the domain migration module and the backbone network by using the HDR video training set, and obtaining the video quality evaluation model.

4. The method of claim 1, wherein, the domain migration module comprises a plurality of residual blocks, and each residual block comprises a convolution layer, an activation function and a residual connection.

5. The method of claim 1, wherein, performing fine-tuning on the video quality evaluation model by using an HDR video training set, and obtaining the video quality evaluation model, comprises: obtaining training samples and verification samples by using the HDR video training set; performing fine-tuning on the video quality evaluation model by using the training samples; verifying the video quality evaluation model by using the verification samples; after the verification is passed, determining a current video quality evaluation model as the video quality evaluation model.

6. The method of claim 5, wherein, obtaining training samples and verification samples by using the SDR video training set, comprises: randomly extracting frames from SDR videos in the SDR video training set, and obtaining the training samples; fixedly extracting frames from the SDR videos in the SDR video training set, and obtaining the verification samples.

7. The method according to any one of claims 1 to 6, characterized in that, performing mapping processing on the target video by using a domain migration module in a trained video quality evaluation model, performing evaluation processing on a video obtained by the mapping by using a backbone model of the video quality evaluation model, and obtaining a quality evaluation result, comprises: extracting a current frame by frame by frame from a reference video and a distortion video corresponding to the target video; mapping high dynamic range features of the current frame to a standard dynamic range domain by using the domain migration module; extracting image features, fusing features and quality regression processing by using the backbone network on the mapped image features to a standard dynamic range domain, to obtain a single frame evaluation result of the current frame; aggregating single frame evaluation results corresponding to each frame of the target video to obtain the quality evaluation result.

8. A video quality determination apparatus characterized by comprising: It comprises: a video acquisition module for acquiring a target video and determining the dynamic range of the target video; wherein the dynamic range includes HDR and SDR, HDR is high dynamic range, and SDR is standard dynamic range; an HDR evaluation module for performing mapping processing on the target video by using a domain migration module in a trained video quality evaluation model, and performing evaluation processing on the mapped video by using a backbone model of the video quality evaluation model to obtain a quality evaluation result when the dynamic range of the target video is HDR; a model training module comprising a first stage training unit, a second stage training unit and a third stage training unit; the first stage training unit is used for quality evaluation training of the backbone network by using an SDR video training set; the second stage training unit is used for adding the domain migration module on the basis of the backbone network after the training of the backbone network is completed; during the quality evaluation training of the video quality evaluation model by using a mixed video set including SDR video and HDR video, if the current video is an HDR video, the current HDR video is mapped and processed by using the domain migration module, and the mapped video is processed by using the backbone network; the third stage training unit is used for fine-tuning the video quality evaluation model by using an HDR video training set to obtain the video quality evaluation model.

9. An electronic device, comprising: It comprises: a memory for storing a computer program; a processor for executing the computer program to realize the steps of the video quality determination method according to any one of claims 1 to 7.

10. A readable storage medium, characterized by, The computer program is stored on the readable storage medium, and the computer program is executed by the processor to realize the steps of the video quality determination method according to any one of claims 1 to 7.

Images