Training method and device of video quality recognition model, video quality recognition method and device

Abstract

The present disclosure relates to a method for training a video quality identification model, a video quality identification method, an apparatus, an electronic device, a storage medium, and a computer program product. The method comprises: obtaining a sample video and a video quality label corresponding to the sample video; inputting the sample video into a pre-trained model combination and a video quality identification model to be trained to obtain a first video quality feature, a first video quality prediction result, a second video quality feature, and a second video quality prediction result of the sample video; training the video quality identification model to be trained according to a difference between the first video quality prediction result and the video quality label, a difference between the second video quality prediction result and the video quality label, and a difference between the first video quality feature and the second video quality feature, to obtain a trained video quality identification model. The method can improve the accuracy of video quality identification.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of video processing, and particularly relates to a training method of a video quality identification model, a video quality identification method and device, electronic equipment, a storage medium and a computer program product. BACKGROUND

[0002] With the development of video processing technology, a technology of identifying video quality to filter out videos with low quality and recommend videos with high quality has appeared. The identification of video quality is mainly to identify low-quality features of the video.

[0003] In the related art, the current video quality identification method is mainly realized by a trained video quality identification model. During training, the model usually fine-tunes a certain pre-trained model so that the fine-tuned model can predict the video quality. The fine-tuned model is then used as the trained video quality identification model. However, the low-quality features of the video include many types, and the function of the pre-trained model is limited, which can only focus on a certain specific low-quality feature of the video. As a result, the video quality prediction result output by the fine-tuned model is not accurate enough, and the identification accuracy of the video quality is low. SUMMARY

[0004] The present disclosure provides a training method of a video quality identification model, a video quality identification method and device, electronic equipment, a storage medium and a computer program product to at least solve the problem of low identification accuracy of video quality in the related art. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of the embodiments of the present disclosure, a training method of a video quality identification model is provided, comprising:

[0006] obtaining a sample video and a video quality label corresponding to the sample video;

[0007] inputting the sample video into a pre-trained model combination to obtain first video quality features and a first video quality prediction result of the sample video, and inputting the sample video into a video quality identification model to be trained to obtain second video quality features and a second video quality prediction result of the sample video; the pre-trained model combination comprises at least two target pre-trained models associated with video quality; the first video quality features are obtained according to first target video features of the sample video output by each target pre-trained model, and the first video quality prediction result is obtained according to the first video quality features; the first target video features output by different target pre-trained models are different;

[0008] According to the difference between the first video quality prediction result and the video quality labeling result, the difference between the second video quality prediction result and the video quality labeling result, and the difference between the first video quality feature and the second video quality feature, the video quality recognition model to be trained is trained to obtain a trained video quality recognition model.

[0009] In an exemplary embodiment, inputting the sample video into a target pre-trained model combination to obtain a first video quality feature and a first video quality prediction result of the sample video includes:

[0010] Inputting the sample videos into respective target pre-training models to obtain respective first target video features of the sample videos;

[0011] performing fusion processing on each of the first target video features according to the importance parameters corresponding to each of the first target video features to obtain a first video quality feature of the sample video;

[0012] Classification processing is performed on the first video quality feature to obtain a first video quality classification result of the sample video as the first video quality prediction result.

[0013] In an exemplary embodiment, inputting the sample videos into respective target pre-training models to obtain respective first target video features of the sample videos includes:

[0014] Inputting the sample videos into respective target pre-training models to obtain respective first initial video features of the sample videos;

[0015] Dimension conversion processing is performed on each first initial video feature of the sample video to obtain each first target video feature of the sample video.

[0016] In an exemplary embodiment, before fusing the first target video features according to the importance parameters corresponding to the first target video features to obtain the first video quality feature of the sample video, the method further includes:

[0017] For each first initial video feature of the sample video, input each first initial video feature into a gating network to obtain an importance parameter of a target pre-trained model for outputting each first initial video feature;

[0018] The importance parameter of each target pre-training model is confirmed as the importance parameter of the first target video feature outputted by each target pre-training model.

[0019] In an example embodiment, the separately inputting the sample video into each target pre-training model to obtain each first initial video feature of the sample video comprises:

[0020] For each target pre-training model, in a case where the target pre-training model belongs to a video model, inputting the sample video into the target pre-training model to obtain a first initial video feature of the sample video;

[0021] In a case where the target pre-training model belongs to an image model, inputting each sample video frame of the sample video into the target pre-training model to obtain an image feature of each sample video frame, and performing fusion processing on the image features of each sample video frame to obtain a first initial video feature of the sample video.

[0022] In an example embodiment, the inputting the sample video into the video quality identification model to be trained to obtain a second video quality feature and a second video quality prediction result of the sample video comprises:

[0023] Inputting the sample video into the video quality identification model to be trained to obtain a second initial video feature of the sample video;

[0024] Performing dimension conversion processing on the second initial video feature of the sample video to obtain a second target video feature of the sample video as the second video quality feature of the sample video; the dimension of the second target video feature is different from the dimension of the second initial video feature;

[0025] Performing classification processing on the second video quality feature to obtain a second video quality classification result of the sample video as the second video quality prediction result.

[0026] In an example embodiment, the training the video quality identification model to be trained according to the difference between the first video quality prediction result and the video quality label result, the difference between the second video quality prediction result and the video quality label result, and the difference between the first video quality feature and the second video quality feature to obtain a trained video quality identification model comprises:

[0027] Obtaining a first loss value according to the difference between the first video quality prediction result and the video quality label result, a second loss value according to the difference between the second video quality prediction result and the video quality label result, and a third loss value according to the difference between the first video quality feature and the second video quality feature; and performing fusion processing on the first loss value, the second loss value, and the third loss value to obtain a target loss value;

[0028] According to the target loss value, the video quality recognition model to be trained is trained until a training end condition is reached; and the trained video quality recognition model reaching the training end condition is the trained video quality recognition model.

[0029] In an example embodiment, the method further comprises:

[0030] A pre-training model library is constructed; the pre-training model library comprises a plurality of pre-training models associated with video quality; and structural differences between model structures of each pre-training model satisfy a preset condition.

[0031] For each pre-training model in the pre-training model library, if a video quality recognition effect of a student model obtained by using the pre-training model is higher than a video quality recognition effect of an updated model obtained by using the pre-training model, the pre-training model is taken as a candidate pre-training model; the student model is obtained by training a video quality recognition model to be trained by taking the pre-training model as a teacher model; and the updated model is obtained by training the pre-training model.

[0032] According to the candidate pre-training models, a candidate pre-training model set is constructed.

[0033] From the candidate pre-training model set, the target pre-training model is screened out.

[0034] According to the target pre-training model, the pre-training model combination is constructed.

[0035] In an example embodiment, the screening of the target pre-training model from the candidate pre-training model set comprises:

[0036] From the candidate pre-training model set, K candidate pre-training models with the highest first video quality recognition effects are screened out; and from the K candidate pre-training models, a candidate pre-training model with the highest second video quality recognition effect is screened out as the target pre-training model.

[0037] The first video quality recognition effect is a video quality recognition effect of a first student model obtained by using a candidate pre-training model; the first student model is obtained by training a video quality recognition model to be trained by taking the candidate pre-training model as a teacher model; the second video quality recognition effect is a video quality recognition effect of a second student model obtained by using the screened candidate pre-training model and the screened target pre-training model; the second student model is obtained by training a video quality recognition model to be trained by taking the screened candidate pre-training model and the screened target pre-training model as teacher models; and K is a positive integer.

[0038] In an exemplary embodiment, the method further comprises:

[0039] When the number of the target pre-trained models is less than N, deleting the target pre-trained model from the candidate pre-trained model set to obtain an updated candidate pre-trained model set;

[0040] The updated candidate pre-training model set is used as the candidate pre-training model set, and the step of screening out K candidate pre-training models with the highest first video quality recognition effect from the candidate pre-training model set, and screening out the candidate pre-training model with the highest second video quality recognition effect from the K candidate pre-training models as the target pre-training model is performed until the number of screened target pre-training models is N, where N is a positive integer.

[0041] According to a second aspect of an embodiment of the present disclosure, a video quality identification method is provided, including:

[0042] Get the video to be identified;

[0043] The video to be identified is input into a trained video quality identification model to obtain a video quality prediction result of the video to be identified; the trained video quality identification model is obtained by training the video quality identification model to be trained using a sample video, the video quality labeling result of the sample video, a first video quality feature and a first video quality prediction result; the first video quality feature is obtained based on the first target video feature of the sample video outputted by at least two target pre-training models associated with video quality, and the first video quality prediction result is obtained based on the first video quality feature; the first target video features outputted by different target pre-training models are different.

[0044] In an exemplary embodiment, inputting the to-be-identified video into a trained video quality recognition model to obtain a video quality prediction result of the to-be-identified video includes:

[0045] Inputting the video to be identified into a trained video quality recognition model to obtain initial video features of the video to be identified;

[0046] Performing dimension conversion on the initial video features to obtain target video features of the video to be identified as video quality features of the video to be identified;

[0047] The video quality features are classified to obtain a video quality classification result of the video to be identified as the video quality prediction result.

[0048] According to a third aspect of an embodiment of the present disclosure, a training device for a video quality recognition model is provided, comprising:

[0049] A sample acquisition unit is configured to acquire a sample video and a video quality labeling result corresponding to the sample video;

[0050] The sample processing unit is configured to input the sample video into a pre-training model combination to obtain a first video quality feature and a first video quality prediction result of the sample video, and input the sample video into a video quality recognition model to be trained to obtain a second video quality feature and a second video quality prediction result of the sample video; the pre-training model combination includes at least two target pre-training models associated with video quality; the first video quality feature is obtained based on the first target video feature of the sample video output by each target pre-training model, and the first video quality prediction result is obtained based on the first video quality feature; the first target video features output by different target pre-training models are different;

[0051] The model training unit is configured to train the video quality recognition model to be trained based on the difference between the first video quality prediction result and the video quality labeling result, the difference between the second video quality prediction result and the video quality labeling result, and the difference between the first video quality feature and the second video quality feature to obtain a trained video quality recognition model.

[0052] In an exemplary embodiment, the sample processing unit is further configured to input the sample video into each target pre-training model respectively to obtain each first target video feature of the sample video; perform fusion processing on each first target video feature according to the importance parameters of the each first target video feature to obtain the first video quality feature of the sample video; perform classification processing on the first video quality feature to obtain the first video quality classification result of the sample video as the first video quality prediction result.

[0053] In an exemplary embodiment, the sample processing unit is further configured to input the sample video into each target pre-training model respectively to obtain each first initial video feature of the sample video; and perform dimensionality conversion processing on each first initial video feature of the sample video to obtain each first target video feature of the sample video.

[0054] In an exemplary embodiment, the device also includes a parameter determination unit, which is configured to execute, for each first initial video feature of the sample video, input each first initial video feature into the gating network, and obtain an importance parameter of the target pre-training model for outputting each first initial video feature; and confirm the importance parameter of each target pre-training model as the importance parameter of the first target video feature corresponding to the output of each target pre-training model.

[0055] In an exemplary embodiment, the sample processing unit is further configured to execute for each target pre-training model, when the target pre-training model belongs to a video model, input the sample video into the target pre-training model to obtain the first initial video feature of the sample video; when the target pre-training model belongs to an image model, input each sample video frame of the sample video into the target pre-training model to obtain the image feature of each sample video frame, and perform fusion processing on the image features of each sample video frame to obtain the first initial video feature of the sample video.

[0056] In an exemplary embodiment, the sample processing unit is further configured to input the sample video into a video quality recognition model to be trained to obtain a second initial video feature of the sample video; perform dimension conversion processing on the second initial video feature of the sample video to obtain a second target video feature of the sample video as a second video quality feature of the sample video; the dimension of the second target video feature is different from the dimension of the second initial video feature; perform classification processing on the second video quality feature to obtain a second video quality classification result of the sample video as the second video quality prediction result.

[0057] In an exemplary embodiment, the model training unit is further configured to obtain a first loss value based on the difference between the first video quality prediction result and the video quality labeling result, obtain a second loss value based on the difference between the second video quality prediction result and the video quality labeling result, and obtain a third loss value based on the difference between the first video quality feature and the second video quality feature; fuse the first loss value, the second loss value and the third loss value to obtain a target loss value; train the video quality recognition model to be trained according to the target loss value until the training end condition is met; the trained video quality recognition model that meets the training end condition is the trained video quality recognition model.

[0058] In an exemplary embodiment, the device also includes a model screening unit, which is configured to execute the construction of a pre-training model library; the pre-training model library includes multiple pre-training models associated with video quality; the structural difference between the model structures of each pre-training model meets a preset condition; for each pre-training model in the pre-training model library, when the video quality recognition effect of the student model obtained using the pre-training model is higher than the video quality recognition effect of the updated model obtained using the pre-training model, the pre-training model is used as a candidate pre-training model; wherein the student model is obtained by training the video quality recognition model to be trained with the pre-training model as the teacher model, and the updated model is obtained by training the pre-training model; based on the candidate pre-training models, a candidate pre-training model set is constructed; from the candidate pre-training model set, the target pre-training model is screened out; based on the target pre-training model, the pre-training model combination is constructed.

[0059] In an exemplary embodiment, the model screening unit is further configured to screen out K candidate pre-training models with the highest first video quality recognition effect from the candidate pre-training model set, and screen out a candidate pre-training model with the highest second video quality recognition effect from the K candidate pre-training models as the target pre-training model; wherein the first video quality recognition effect is the video quality recognition effect of the first student model obtained using the candidate pre-training model; the first student model is obtained by training the video quality recognition model to be trained with the candidate pre-training model as the teacher model; the second video quality recognition effect is the video quality recognition effect of the second student model obtained by using the screened candidate pre-training model and the screened target pre-training model; the second student model is obtained by training the video quality recognition model to be trained with the screened candidate pre-training model and the screened target pre-training model as the teacher model; K is a positive integer.

[0060] In an exemplary embodiment, the model screening unit is further configured to execute the step of deleting the target pre-trained model from the candidate pre-trained model set when the number of the target pre-trained models is less than N, to obtain an updated candidate pre-trained model set; using the updated candidate pre-trained model set as the candidate pre-trained model set, and jumping to the step of screening out the K candidate pre-trained models with the highest first video quality recognition effect from the candidate pre-trained model set, and screening out the candidate pre-trained model with the highest second video quality recognition effect from the K candidate pre-trained models as the target pre-trained model, until the number of screened target pre-trained models is N; N is a positive integer.

[0061] According to a fourth aspect of an embodiment of the present disclosure, a video quality identification device is provided, including:

[0062] A video acquisition unit is configured to acquire a video to be identified;

[0063] The quality identification unit is configured to execute the input of the video to be identified into the trained video quality identification model to obtain the video quality prediction result of the video to be identified; the trained video quality identification model is obtained by training the video quality identification model to be trained using a sample video, the video quality labeling result of the sample video, a first video quality feature and a first video quality prediction result; the first video quality feature is obtained based on the first target video feature of the sample video outputted by at least two target pre-training models associated with video quality, and the first video quality prediction result is obtained based on the first video quality feature; the first target video features outputted by different target pre-training models are different.

[0064] In an exemplary embodiment, the quality identification unit is further configured to execute the input of the video to be identified into a trained video quality identification model to obtain the initial video features of the video to be identified; perform dimensionality conversion processing on the initial video features to obtain the target video features of the video to be identified as the video quality features of the video to be identified; perform classification processing on the video quality features to obtain the video quality classification results of the video to be identified as the video quality prediction results.

[0065] According to a fifth aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0066] processor;

[0067] a memory for storing instructions executable by the processor;

[0068] The processor is configured to execute the instructions to implement the video quality recognition model training method described in any one of the embodiments of the first aspect, or the video quality recognition method described in any one of the embodiments of the second aspect.

[0069] According to a sixth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the video quality recognition model as described in any embodiment of the first aspect, or the video quality recognition method as described in any embodiment of the second aspect.

[0070] According to a seventh aspect of the embodiments of the present disclosure, a computer program product is provided, which includes instructions. When the instructions are executed by a processor of an electronic device, the electronic device is capable of executing the training method of the video quality recognition model as described in any embodiment of the first aspect, or the video quality recognition method as described in any embodiment of the second aspect.

[0071] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0072] By obtaining a sample video and a video quality labeling result corresponding to the sample video; then inputting the sample video into a pre-training model combination to obtain a first video quality feature and a first video quality prediction result of the sample video, and inputting the sample video into a video quality recognition model to be trained to obtain a second video quality feature and a second video quality prediction result of the sample video; the pre-training model combination includes at least two target pre-training models associated with video quality; finally, according to the difference between the first video quality prediction result and the video quality labeling result, the difference between the second video quality prediction result and the video quality labeling result, and the difference between the first video quality feature and the second video quality feature, the video quality recognition model to be trained is trained to obtain a trained video quality recognition model. In this way, during the model training process, multiple target pre-training models associated with video quality are used to assist in the training of the video quality recognition model. Since the first target video features output by different target pre-training models are different, it means that different target pre-training models focus on different types of low-quality features. Then, the trained video quality recognition model can perceive multiple types of low-quality features, so that the trained video quality recognition model can recognize video quality more comprehensively, thereby improving the accuracy of the output video quality prediction results, which is beneficial to improving the recognition accuracy of video quality; at the same time, the first video quality features, first video quality prediction results and video quality labeling results of the sample videos jointly output by multiple target pre-training models associated with video quality are used as supervision information to assist in the training of the video quality recognition model. Taking into account multiple supervision information, the prediction results of the trained video quality recognition model are more accurate, further improving the recognition accuracy of video quality.

[0073] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0075] Figure 1 The figure is a flowchart of a method for training a video quality recognition model according to an exemplary embodiment.

[0076] Figure 2 The flowchart shows the steps of obtaining a first video quality feature and a first video quality prediction result of a sample video according to an exemplary embodiment.

[0077] Figure 3 is a schematic diagram of training a video quality recognition model according to an exemplary embodiment;

[0078] Figure 4 The flowchart shows the steps of obtaining the second video quality feature and the second video quality prediction result of the sample video according to an exemplary embodiment.

[0079] Figure 5 The figure is a flowchart showing steps of screening a target pre-training model according to an exemplary embodiment.

[0080] Figure 6 The figure is a flowchart of another method for training a video quality recognition model according to an exemplary embodiment.

[0081] Figure 7 The figure is a flowchart of a video quality identification method according to an exemplary embodiment.

[0082] Figure 8 FIG. 1 is a schematic diagram showing comparison of video quality prediction results according to an exemplary embodiment.

[0083] Figure 9 The figure is a block diagram of a video quality recognition model training device according to an exemplary embodiment.

[0084] Figure 10 The figure is a block diagram of a video quality identification device according to an exemplary embodiment.

[0085] Figure 11 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0086] In order to enable ordinary people in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0087] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0088] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0089] Figure 1 FIG. 1 is a flow chart showing a method for training a video quality recognition model according to an exemplary embodiment. Figure 1 As shown, the training method of the video quality recognition model is used in a terminal; it is understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and implemented through interaction between the terminal and the server. In this exemplary embodiment, the method includes the following steps:

[0090] In step S110, a sample video and a video quality labeling result corresponding to the sample video are obtained.

[0091] Among them, sample videos refer to videos used to train video quality recognition models, such as videos shot by users, videos on the Internet, etc.

[0092] Each sample video is matched with a corresponding video quality annotation result. The video quality annotation result refers to the video quality annotation score, such as 4 or 5, obtained through manual annotation. In actual scenarios, the video quality annotation result refers to the Mean Opinion Score (MOS).

[0093] Specifically, the terminal obtains the sample video and the video quality labeling results corresponding to the sample video from the local database, so as to facilitate subsequent use of the sample video and the video quality labeling results corresponding to the sample video to train a video quality recognition model.

[0094] In step S120, the sample video is input into the pre-trained model combination to obtain the first video quality feature and the first video quality prediction result of the sample video, and the sample video is input into the video quality recognition model to be trained to obtain the second video quality feature and the second video quality prediction result of the sample video; the pre-trained model combination includes at least two target pre-trained models associated with video quality.

[0095] Among them, the first video quality feature is obtained according to the first target video feature of the sample video output corresponding to each target pre-training model, and the first video quality prediction result is obtained according to the first video quality feature; the first target video features output by different target pre-training models are different.

[0096] Each target pre-training model is used to output a first target video feature corresponding to a sample video; the first video quality feature is obtained by fusing the first target video features of the sample video output by each target pre-training model; and the first video quality prediction result is obtained by classifying the first video quality features. Furthermore, since different target pre-training models output different first target video features, this indicates that different target pre-training models focus on different types of low-quality features.

[0097] The pre-training model combination includes multiple target pre-training models related to video quality, such as pre-training models based on video classification, pre-training models based on image classification, and pre-training models based on visual text tasks. Each target pre-training model focuses on different types of low-quality video features (such as edge blur, block effects, etc.). For example, some target pre-training models focus on edge blur, while some target pre-training models focus on block effects. A target pre-training model associated with video quality means that the target pre-training model can be applied to video quality recognition tasks.

[0098] Among them, each target pre-training model refers to the teacher model in knowledge distillation, and the pre-training model combination refers to the combination of multiple teacher models. The combination of multiple teacher models can achieve better knowledge transfer effect on the video quality recognition model to be trained.

[0099] It should be noted that each target pre-training model achieves good results in its original respective field, such as high accuracy in image classification tasks and high accuracy in video classification tasks. Moreover, each target pre-training model varies greatly, including in terms of model structure, pre-training dataset, and number of parameters. This ensures that each target pre-training model has its own area of ​​expertise and that different target pre-training models can complement each other. For example, a pre-training model based on video classification can effectively model spatiotemporal relationships, a pre-training model based on image classification can perceive content-related features, and a pre-training model based on visual text tasks can capture human emotional expressions. These differences can provide a more comprehensive feature representation for video quality recognition tasks.

[0100] Among them, the target pre-training models in the pre-training model combination are divided into two types. One is the video model (i.e., the model for processing videos), such as video classification model, video detection model, etc. In actual scenarios, the video model can refer to SlowFast (video target detection model), Ir-CSN-152 (cognitive transfer neuron model), etc. The other is the image model (i.e., the model for processing images), such as image classification model, image detection model, etc. In actual scenarios, the image model can refer to EfficientNet (fast and high-precision model), CLIP (multimodal model), etc. Figure 3 , the teacher model includes a video model and an image model. The video model includes a pre-trained model V1...pre-trained model Vn, and the image model includes a pre-trained model M1...pre-trained model Mn. It should be noted that, Figure 3 The pre-trained models mentioned here refer to target pre-trained models associated with video quality.

[0101] Among them, the video quality recognition model to be trained refers to the student model in knowledge distillation, such as Video SwinTiny (video classification model), Ip-CSN-152 (cognitive transfer neuron model), etc.

[0102] The first video quality feature refers to the feature information related to the video quality of the sample video obtained by combining the pre-trained models, such as Figure 3 The Zt in the example is specifically composed of the video features extracted from the sample video by each target pre-training model (e.g. Figure 3 Since each target pre-training model focuses on different types of low-quality video features, the fused first video quality feature can cover multiple types of low-quality video features.

[0103] The first video quality prediction result refers to the prediction result reflected by the first video quality feature, which is obtained by performing full connection processing on the first video quality feature, such as Figure 3 The video quality prediction result output by the teacher model in [1]. In actual scenarios, the first video quality prediction result refers to the first video quality prediction score, such as 4 points.

[0104] The second video quality feature refers to feature information related to the video quality of the sample video obtained by the video quality recognition model to be trained, such as Figure 3 The Zs in the video quality recognition model to be trained is specifically the video features extracted from the sample video (such as Figure 3 The Vs) in the equation is obtained by processing.

[0105] The second video quality prediction result refers to the prediction result reflected by the second video quality feature, which is obtained by performing full connection processing on the second video quality feature, such as Figure 3 The video quality prediction result output by the student model in [1]. In actual scenarios, the second video quality prediction result refers to the second video quality prediction score, such as 3 points.

[0106] Specifically, the terminal inputs the sample video into each target pre-training model in the pre-training model combination, extracts corresponding video features from the sample video through each target pre-training model, and fuses the various video features to obtain a first video quality feature; performs full-connection processing on the first video quality feature to obtain a first video quality prediction result for the sample video. Simultaneously, the terminal inputs the sample video into the video quality recognition model to be trained, extracts corresponding video features from the sample video through the video quality recognition model to be trained as a second video quality feature; performs full-connection processing on the second video quality feature to obtain a second video quality prediction result for the sample video.

[0107] In step S130, the video quality recognition model to be trained is trained based on the difference between the first video quality prediction result and the video quality labeling result, the difference between the second video quality prediction result and the video quality labeling result, and the difference between the first video quality feature and the second video quality feature to obtain a trained video quality recognition model.

[0108] Specifically, the terminal obtains a target loss value based on the difference between the first video quality prediction result and the video quality labeling result, the difference between the second video quality prediction result and the video quality labeling result, and the difference between the first video quality feature and the second video quality feature; according to the target loss value, the video quality recognition model to be trained is trained until the training end condition is met; the trained video quality recognition model that meets the training end condition is confirmed as the trained video quality recognition model.

[0109] It should be noted that the training end condition refers to the current training times reaching the preset training times, the current target loss value being less than the preset threshold, etc.

[0110] In the training method of the above-mentioned video quality recognition model, a sample video and a video quality labeling result corresponding to the sample video are obtained; then the sample video is input into the pre-training model combination to obtain the first video quality feature and the first video quality prediction result of the sample video, and the sample video is input into the video quality recognition model to be trained to obtain the second video quality feature and the second video quality prediction result of the sample video; the pre-training model combination includes at least two target pre-training models associated with video quality; finally, according to the difference between the first video quality prediction result and the video quality labeling result, the difference between the second video quality prediction result and the video quality labeling result, and the difference between the first video quality feature and the second video quality feature, the video quality recognition model to be trained is trained to obtain a trained video quality recognition model. In this way, during the model training process, multiple target pre-training models associated with video quality are used to assist in the training of the video quality recognition model. Since the first target video features output by different target pre-training models are different, it means that different target pre-training models focus on different types of low-quality features. Then, the trained video quality recognition model can perceive multiple types of low-quality features, so that the trained video quality recognition model can recognize video quality more comprehensively, thereby improving the accuracy of the output video quality prediction results, which is beneficial to improving the recognition accuracy of video quality; at the same time, the first video quality features, first video quality prediction results and video quality labeling results of the sample videos jointly output by multiple target pre-training models associated with video quality are used as supervision information to assist in the training of the video quality recognition model. Taking into account multiple supervision information, the prediction results of the trained video quality recognition model are more accurate, further improving the recognition accuracy of video quality.

[0111] In an exemplary embodiment, Figure 2 As shown, in step S120, the sample video is input into the target pre-trained model combination to obtain the first video quality feature and the first video quality prediction result of the sample video, which can be specifically achieved by the following steps:

[0112] In step S210, the sample videos are input into respective target pre-training models to obtain respective first target video features of the sample videos.

[0113] Among them, the first target video feature refers to the video feature obtained after the video feature output by the target pre-training model is converted into a feature dimension, such as Figure 3 Zi, Zj, etc.

[0114] In step S220, the first target video features are fused according to the importance parameters of the first target video features to obtain the first video quality features of the sample video.

[0115] The importance parameter of the first target video feature refers to the weight of the first target video feature, which is used to measure the importance of the first target video feature; different first target video features correspond to different weights.

[0116] The fusion processing refers to weighted summation processing.

[0117] In step S230, the first video quality feature is classified to obtain a first video quality classification result of the sample video as a first video quality prediction result.

[0118] The classification processing refers to a fully connected processing; the first video quality classification result is used to represent a classification result of the video quality of the sample video, such as low quality, high quality, etc.

[0119] Specifically, the terminal inputs the sample video into each target pre-training model respectively, performs feature extraction processing on the sample video through each target pre-training model to obtain each first target video feature of the sample video; obtains the importance parameters corresponding to each first target video feature, and fuses each first target video feature according to the importance parameters corresponding to each first target video feature to obtain the first video quality feature of the sample video; for example, performs weighted summation processing on each first target video feature according to the weight corresponding to each first target video feature to obtain the first video quality feature of the sample video; classifies the first video quality feature to obtain the corresponding first video quality classification result as the first video quality prediction result of the sample video; for example, performs full connection processing on the first video quality feature to obtain the first video quality classification result of the sample video.

[0120] For example, refer to Figure 3 , the terminal inputs the sample video into each target pre-training model, such as pre-training model V1...pre-training model Vn, pre-training model M1...pre-training model Mn; performs feature extraction processing on the sample video through each target pre-training model to obtain each first target video feature of the sample video, such as Zi, Zj, etc.; obtains the weight corresponding to each first target video feature, such as Wi, Wj, etc., and performs weighted summation processing on each first target video feature according to the weight corresponding to each first target video feature to obtain the first video quality feature of the sample video, such as Zt=Zi×Wi+…+Zj×Wj; inputs the first video quality feature Zt into the fully connected layer, performs full connection processing on the first video quality feature Zt through the fully connected layer, and obtains the first video quality prediction result.

[0121] The technical solution provided by the embodiment of the present disclosure inputs the sample video into each target pre-training model respectively to obtain each first target video feature of the sample video; according to the importance parameters corresponding to each first target video feature, each first target video feature is fused to obtain the first video quality feature of the sample video; since different target pre-training models focus on different low-quality video features, the obtained first video quality feature can cover multiple types of low-quality video features, so that the video quality recognition model trained based on the first video quality feature can also focus on multiple types of low-quality video features, thereby improving the recognition accuracy of video quality. At the same time, the first video quality feature is classified to obtain the first video quality prediction result of the sample video, which is beneficial to use the first video quality prediction result as a kind of supervisory information in the subsequent process of training the video quality recognition model to assist model training, thereby further improving the video quality prediction accuracy of the video quality recognition model.

[0122] In an exemplary embodiment, in step S210, the sample video is input into each target pre-training model respectively to obtain each first target video feature of the sample video, which specifically includes the following contents: the sample video is input into each target pre-training model respectively to obtain each first initial video feature of the sample video; and the each first initial video feature of the sample video is dimensionally converted to obtain each first target video feature of the sample video.

[0123] Among them, the first initial video feature refers to the video feature output by the target pre-training model, such as Figure 3 Vi, Vj, etc.

[0124] Among them, the dimension conversion processing refers to converting the feature dimension of the first initial video feature, such as converting the feature dimension of the first initial video feature from dimension A (such as 20 dimensions) to dimension B (such as 10 dimensions), which can be obtained through full connection processing and normalization processing.

[0125] Among them, the feature dimensions corresponding to each first target video feature are the same.

[0126] It should be noted that the feature dimensions of the first initial video features output by different target pre-training models are different. In order to ensure that the feature dimensions of the first initial video features extracted by different target pre-training models can match, it is necessary to convert the feature dimensions of the first initial video features output by each target pre-training model so that they are all mapped to the same feature dimension.

[0127] Specifically, the terminal inputs the sample video into each target pre-training model respectively, performs feature extraction processing on the sample video through each target pre-training model, and obtains each first initial video feature of the sample video; inputs each first initial video feature of the sample video into the fully connected layer and the normalization layer respectively, performs dimension conversion processing on each first initial video feature through the fully connected layer and the normalization layer, and obtains each dimension-converted video feature of the sample video, which corresponds to each first target video feature of the sample video.

[0128] For example, refer to Figure 3 , the terminal inputs the sample video into each target pre-training model, such as pre-training model V1...pre-training model Vn, pre-training model M1...pre-training model Mn; performs feature extraction processing on the sample video through each target pre-training model to obtain each first initial video feature of the sample video, such as Vi, Vj, etc.; inputs each first initial video feature into the FC layer (fully connected layer) and the LN layer (normalization layer), performs feature dimension conversion on the first initial video feature through the FC layer and the LN layer, and obtains each first target video feature of the sample video, such as Zi, Zj, etc.

[0129] The technical solution provided by the embodiment of the present disclosure inputs the sample video into each target pre-training model respectively to obtain each first initial video feature of the sample video; performs dimension conversion processing on each first initial video feature of the sample video to obtain each first target video feature of the sample video, and fully considers the first target video features obtained by each target pre-training model, which is conducive to the subsequent fusion of each first target video feature to obtain the first video quality feature, and trains the video quality recognition model based on the first video quality feature, so that the trained video quality recognition model can also pay attention to multiple types of low-quality video features, avoiding the defect of only paying attention to a specific low-quality video feature, resulting in low video quality recognition accuracy, and further improving the video quality recognition accuracy.

[0130] In an exemplary embodiment, in step S220, before fusing each first target video feature according to its importance parameter to obtain the first video quality feature of the sample video, the following contents are also included: for each first initial video feature of the sample video, each first initial video feature is input into the gating network to obtain the importance parameter of the target pre-training model for outputting each first initial video feature; the importance parameter of each target pre-training model is confirmed as the importance parameter of the first target video feature corresponding to the output of each target pre-training model.

[0131] The gated network is used to output the importance parameters (such as weights) of the target pre-trained model for outputting each first initial video feature according to each first initial video feature of the input sample video. The gated network is composed of a fully connected layer and a normalized layer, such as Figure 3 In

[15] , the gating network consists of FC layer (fully connected layer) and SoftMax (normalization layer).

[0132] Among them, the importance parameter corresponding to the target pre-training model refers to the weight corresponding to the target pre-training model, which is used to measure the importance of the target pre-training model; different target pre-training models correspond to different weights.

[0133] The importance parameter of the first target video feature refers to the weight of the first target video feature, which is used to measure the importance of the first target video feature; the first target video features output by different target pre-training models correspond to different weights.

[0134] Specifically, the terminal splices the first initial video features of the sample video to obtain spliced ​​video features; inputs the spliced ​​video features into the gated network, and performs full connection processing and normalization processing on the spliced ​​video features through the gated network to obtain the importance parameters (such as weights) of each target pre-trained model used to output each first initial video feature; the importance parameters corresponding to each target pre-trained model are corresponded to the importance parameters of the first target video features output by each target pre-trained model.

[0135] For example, refer to Figure 3 The terminal splices the first initial video features (such as Vi, Vj, etc.) of the sample video and inputs them into the gating network. The spliced ​​video features are fully connected and normalized through the FC layer and SoftMax layer in the gating network to obtain the weights of the target pre-trained models used to output the first initial video features, which correspond to the weights of the first target video features.

[0136] The technical solution provided by the embodiment of the present disclosure inputs each first initial video feature of the sample video into the gating network to obtain the importance parameter of the target pre-training model used to output each first initial video feature; the importance parameter of each target pre-training model is corresponded to the importance parameter of the first target video feature outputted by each target pre-training model; in this way, by introducing the gating network, the dynamic weight corresponding to each first target video feature is confirmed, and the dynamic weight corresponding to each first target video feature is comprehensively considered, which is conducive to improving the accuracy of the first video quality feature subsequently obtained based on the fusion of each first target video feature, and further improving the prediction accuracy of the video quality recognition model trained based on the first video quality feature.

[0137] In an exemplary embodiment, the sample video is input into each target pre-training model respectively to obtain each first initial video feature of the sample video, which specifically includes the following contents: for each target pre-training model, when the target pre-training model is a video model, the sample video is input into the target pre-training model to obtain the first initial video feature of the sample video; when the target pre-training model is an image model, each sample video frame of the sample video is input into the target pre-training model to obtain the image feature of each sample video frame, and the image features of each sample video frame are fused to obtain the first initial video feature of the sample video.

[0138] The target pre-training model belongs to the video model, which means that the target pre-training model is Figure 3 The video models shown, such as video classification models, video detection models, etc., are used to process videos and can directly output feature information of the videos.

[0139] Among them, the target pre-training model belongs to the image model, which means that the target pre-training model is Figure 3 The image models shown, such as image classification models, image detection models, etc., are used to process images and can directly output feature information of the images.

[0140] The sample video frames of the sample video refer to the video frames obtained after the sample video is divided into frames, such as Figure 3 Shown are video frame 1, video frame 2, video frame 3, ... video frame n.

[0141] Specifically, the terminal identifies the model type of each target pre-training model respectively. When the target pre-training model is a video model, the sample video is directly input into the target pre-training model, and the target pre-training model is used to perform feature extraction on the sample video to obtain the first initial video feature of the sample video; when the target pre-training model is an image model, the sample video is frame-processed to obtain each sample video frame of the sample video, and each sample video frame is input into the target pre-training model respectively, and the target pre-training model is used to perform feature extraction on each sample video frame to obtain the image feature corresponding to each sample video frame; the image features corresponding to each sample video frame are averaged and fused to obtain the first initial video feature of the sample video.

[0142] For example, refer to Figure 3If the terminal recognizes that the target pre-training model is a video model, the sample video is directly input into the target pre-training model (such as pre-training model V1), and the first initial video feature of the sample video, such as Vi, is output through the target pre-training model; if the terminal recognizes that the target pre-training model is an image model, each sample video frame of the sample video (such as video frame 1, video frame 2, etc.) is input into the target pre-training model (such as pre-training model M1) respectively, and the image features of each sample video frame are output through the target pre-training model, and the image features of each sample video frame are averaged and fused to obtain the first initial video feature of the sample video, such as Vj.

[0143] The technical solution provided by the embodiment of the present disclosure distinguishes whether the target pre-trained model is a video model or an image model, and processes the sample video differently, which is beneficial to improving the extraction accuracy of the first initial video feature.

[0144] In an exemplary embodiment, Figure 4 As shown, in step S120, the sample video is input into the video quality recognition model to be trained to obtain the second video quality feature and the second video quality prediction result of the sample video, which can be specifically achieved by the following steps:

[0145] In step S410, the sample video is input into the video quality recognition model to be trained to obtain the second initial video features of the sample video.

[0146] The second initial video feature refers to the video feature output by the video quality recognition model to be trained, such as Figure 3 Vs.

[0147] In step S420, the second initial video feature of the sample video is dimensionally transformed to obtain a second target video feature of the sample video as a second video quality feature of the sample video.

[0148] The dimension of the second target video feature is different from the dimension of the second initial video feature.

[0149] The second target video feature refers to the video feature output by the video quality recognition model to be trained (such as the second initial video feature) after the feature dimension conversion, such as Figure 3 The Zs in .

[0150] Among them, the dimension conversion processing refers to converting the feature dimension of the second initial video feature, such as converting the feature dimension of the second initial video feature from dimension A (such as 20 dimensions) to dimension B (such as 10 dimensions), which can be obtained through full connection processing and normalization processing.

[0151] It should be noted that the dimension conversion processing is performed to ensure that the feature dimension of the second target video feature is the same as the feature dimension of the first target video feature.

[0152] In step S430, the second video quality feature is classified to obtain a second video quality classification result of the sample video as a second video quality prediction result.

[0153] The classification processing refers to full connection processing, and the second video quality classification result is used to represent a classification result of the video quality of the sample video, such as low quality, high quality, etc.

[0154] Specifically, the terminal inputs the sample video into the video quality recognition model to be trained, performs feature extraction processing on the sample video through the video quality recognition model to be trained to obtain a second initial video feature of the sample video; inputs the second initial video feature of the sample video into a full connection layer and a normalization layer, and performs dimension conversion processing on the second initial video feature through the full connection layer and the normalization layer to obtain a converted video feature of the sample video as a second target video feature of the sample video; confirms the second target video feature of the sample video as a second video quality feature of the sample video; inputs the second video quality feature into the full connection layer, and performs full connection processing on the second video quality feature through the full connection layer to classify the second video quality feature to obtain a corresponding second video quality classification result as a second video quality prediction result of the sample video.

[0155] For example, referring to Figure 3 , the terminal inputs the sample video into the video quality recognition model to be trained, performs feature extraction processing on the sample video through the video quality recognition model to be trained to obtain a second initial video feature of the sample video, such as Vs; inputs the second initial video feature Vs into a FC layer (full connection layer) and a LN layer (normalization layer), and performs dimension conversion processing on the second initial video feature through the FC layer and the LN layer to obtain a second target video feature of the sample video, such as Zs; confirms the second target video feature Zs of the sample video as a second video quality feature of the sample video; inputs the second video quality feature Zs into the full connection layer, and performs full connection processing on the second video quality feature Zs through the full connection layer to obtain a second video quality classification result of the sample video as a second video quality prediction result of the sample video.

[0156] The technical solution provided by the embodiment of the present disclosure inputs a sample video into a video quality recognition model to be trained to obtain a second video quality feature and a second video quality prediction result of the sample video, which is conducive to subsequently training a video quality recognition model based on the second video quality feature and the second video quality prediction result of the sample video. Comprehensive consideration of the second video quality feature and the second video quality prediction result of the sample video is conducive to improving the video quality prediction accuracy of the trained video quality recognition model.

[0157] In an exemplary embodiment, in step S130, the video quality recognition model to be trained is trained based on the difference between the first video quality prediction result and the video quality labeling result, the difference between the second video quality prediction result and the video quality labeling result, and the difference between the first video quality feature and the second video quality feature to obtain a trained video quality recognition model, which specifically includes the following contents: obtaining a first loss value based on the difference between the first video quality prediction result and the video quality labeling result, obtaining a second loss value based on the difference between the second video quality prediction result and the video quality labeling result, and obtaining a third loss value based on the difference between the first video quality feature and the second video quality feature; fusing the first loss value, the second loss value and the third loss value to obtain a target loss value; training the video quality recognition model to be trained based on the target loss value until a training end condition is met; the trained video quality recognition model that meets the training end condition is a trained video quality recognition model.

[0158] The fusing of the first loss value, the second loss value, and the third loss value refers to adding the first loss value, the second loss value, and the third loss value.

[0159] Specifically, the terminal obtains a first loss value based on the difference between the first video quality prediction result and the video quality labeling result, and the first loss function; obtains a second loss value based on the difference between the second video quality prediction result and the video quality labeling result, and the second loss function; obtains a third loss value based on the difference between the first video quality feature and the second video quality feature, and the third loss function; fuses the first loss value, the second loss value and the third loss value (for example, adds them together) to obtain a target loss value; adjusts the model parameters of the video quality recognition model to be trained according to the target loss value to obtain a video quality recognition model after the model parameters are adjusted; trains the video quality recognition model after the model parameters are adjusted again until the training end condition is met, and the trained video quality recognition model that meets the training end condition is used as the trained video quality recognition model.

[0160] It should be noted that the first loss function, the second loss function and the third loss function all refer to the mean square error loss function.

[0161] For example, when the target loss value is less than the preset threshold, the terminal adjusts the model parameters of the video quality recognition model to be trained according to the target loss value, and re-trains the video quality recognition model after the model parameters are adjusted until the target loss value obtained according to the trained video quality recognition model is less than the preset threshold, and the trained video quality recognition model is used as the trained video quality recognition model.

[0162] Further, refer to Figure 3 , the target loss value can be calculated by the following formula:

[0163] L=L1(F s (x),y)+w(L1(F t (x),y)+L2(Zs,Zt));

[0164] Where L represents the target loss value, L1 and L2 both represent mean square error loss functions, x represents the sample video, y represents the video quality labeling result of the sample video, Zt represents the first video quality feature of the sample video, and Zs represents the second video quality feature of the sample video; F t (x) represents the first video quality prediction result obtained by the multi-teacher model, F s (x) represents the second video quality prediction result obtained by the student model; w represents the weight, which is generally equal to 4.

[0165] The technical solution provided by the embodiment of the present disclosure trains the video quality recognition model to be trained based on the difference between the first video quality prediction result and the video quality labeling result, the difference between the second video quality prediction result and the video quality labeling result, and the difference between the first video quality feature and the second video quality feature to obtain a trained video quality recognition model; in this way, by comprehensively considering a variety of auxiliary information, the trained video quality recognition model can focus on a variety of different types of low-quality video features, rather than just focusing on a specific low-quality video feature, thereby improving the recognition accuracy of video quality.

[0166] In an exemplary embodiment, Figure 5 As shown, the training method of the video quality recognition model provided by the present disclosure also includes the step of screening the target pre-training model, which can be specifically implemented by the following steps:

[0167] In step S510, a pre-trained model library is constructed; the pre-trained model library includes multiple pre-trained models associated with video quality; the structural differences between the model structures of each pre-trained model meet preset conditions.

[0168] The structural difference between the model structures of each pre-trained model satisfies the preset condition, which means that the structural difference between the model structures of each pre-trained model is large, for example, the structural difference is greater than 50%. It should be noted that, assuming that the structural difference between the model structures of two pre-trained models is greater than 50%, it means that the structural similarity between the model structures of the two pre-trained models is less than 50%.

[0169] Each pre-trained model achieves good results in its respective field. Furthermore, each pre-trained model differs significantly in terms of model structure, pre-trained dataset, and number of parameters. This ensures that each pre-trained model has its own area of ​​expertise and that different pre-trained models can complement each other. These differences provide a more comprehensive feature representation for video quality recognition tasks. Furthermore, each pre-trained model must be suitable for video quality recognition tasks. For example, a student model trained using a pre-trained model must be able to effectively handle video quality recognition tasks.

[0170] In step S520, for each pre-trained model in the pre-trained model library, if the video quality recognition effect of the student model obtained using the pre-trained model is higher than the video quality recognition effect of the updated model obtained using the pre-trained model, the pre-trained model is used as a candidate pre-trained model.

[0171] Among them, the student model is obtained by training the video quality recognition model to be trained with the pre-trained model as the teacher model; the update model is obtained by training the pre-trained model.

[0172] The video quality recognition effect refers to the recognition effect of the video quality, such as the video quality recognition accuracy. Of course, the video quality recognition effect can also be expressed by SRCC (Spearman's Rank Correlation Coefficient).

[0173] Among them, the candidate pre-training model refers to the pre-training model preliminarily screened from the pre-training model library.

[0174] In step S530, a candidate pre-training model set is constructed based on the candidate pre-training models.

[0175] The candidate pre-training model set includes candidate pre-training models.

[0176] In step S540, a target pre-training model is screened out from the candidate pre-training model set.

[0177] Among them, the number of screened target pre-training models is N, and N is a positive integer.

[0178] In step S550, a pre-training model combination is constructed according to the target pre-training model.

[0179] Specifically, the terminal obtains pre-trained models that have achieved good results in their respective fields and are associated with video quality, and constructs a pre-trained model library based on these pre-trained models; for each pre-trained model in the pre-trained model library, the pre-trained model is used as a teacher model to train the video quality recognition model to be trained, and the trained video quality recognition model is obtained as a student model, and the pre-trained model is trained to obtain the trained model as an updated model; the video quality recognition effect of the student model and the updated model on the same test data set (a data set composed of test videos) is obtained, if the video quality recognition effect of the student model is higher than the video quality recognition effect of the update model, the pre-trained model is used as a candidate pre-trained model, thereby screening out candidate pre-trained models from the pre-trained model library; the candidate pre-trained models are combined to obtain a candidate pre-trained model set; a greedy strategy is adopted to screen out N target pre-trained models suitable for the video quality recognition task from the candidate pre-trained model set; these N target pre-trained models are combined to obtain a pre-trained model combination.

[0180] It should be noted that only when the video quality recognition effect of the student model is higher than that of the updated model, the corresponding pre-trained model will be used as a candidate pre-trained model; if the video quality recognition effect of the student model is less than or equal to the video quality recognition effect of the updated model, the corresponding pre-trained model will not be used as a candidate pre-trained model.

[0181] The technical solution provided by the embodiment of the present disclosure first screens out candidate pre-training models from the pre-training model library, combines the candidate pre-training models to obtain a candidate pre-training model set; then screens out target pre-training models from the candidate pre-training model set, and finally combines the target pre-training models to obtain a pre-training model combination; in this way, by selecting multiple optimal target pre-training models from the pre-training model library, the video quality prediction results of the video quality recognition model subsequently trained based on the multiple optimal target pre-training models are more accurate, thereby further improving the accuracy of video quality recognition.

[0182] In an exemplary embodiment, in step S540, a target pre-training model is screened out from the set of candidate pre-training models, specifically including the following contents: K candidate pre-training models with the highest first video quality recognition effect are screened out from the set of candidate pre-training models, and a candidate pre-training model with the highest second video quality recognition effect is screened out from the K candidate pre-training models as the target pre-training model; wherein the first video quality recognition effect is the video quality recognition effect of the first student model obtained by using the candidate pre-training model; the first student model is obtained by training the video quality recognition model to be trained with the candidate pre-training model as the teacher model; the second video quality recognition effect is the video quality recognition effect of the second student model obtained by using the screened candidate pre-training model and the screened target pre-training model; the second student model is obtained by training the video quality recognition model to be trained with the screened candidate pre-training model and the screened target pre-training model as the teacher model; K is a positive integer.

[0183] Furthermore, when the number of target pre-trained models is less than N, the target pre-trained model is deleted from the candidate pre-trained model set to obtain an updated candidate pre-trained model set; the updated candidate pre-trained model set is used as the candidate pre-trained model set, and the process jumps to the step of screening out the K candidate pre-trained models with the highest first video quality recognition effect from the candidate pre-trained model set, and screening out the candidate pre-trained model with the highest second video quality recognition effect from the K candidate pre-trained models as the target pre-trained model, until the number of screened target pre-trained models is N; N is a positive integer.

[0184] The first student model is obtained by training the video quality recognition model to be trained using the candidate pre-trained model as a single teacher model. It should be noted that the candidate pre-trained model here refers to any candidate pre-trained model in the candidate pre-trained model set.

[0185] The second student model is obtained by training the video quality recognition model to be trained using the selected candidate pre-trained models and the selected target pre-trained model as multiple teacher models. It should be noted that the selected candidate pre-trained model refers to any candidate pre-trained model selected from the set of candidate pre-trained models; the selected target pre-trained model refers to all candidate pre-trained models previously selected from the K candidate pre-trained models.

[0186] The first and second video quality recognition effects also refer to the recognition effects of video quality, such as the video quality recognition accuracy. Of course, the first and second video quality recognition effects can also be expressed using the Spearman's Rank Correlation Coefficient (SRCC). It should be noted that both the first and second video quality recognition effects were obtained on the same test dataset (a dataset consisting of test videos).

[0187] It should be noted that the present disclosure does not limit the size relationship between K and N.

[0188] Specifically, the terminal uses each candidate pre-trained model in the candidate pre-trained model set as a teacher model to train the video quality recognition model to be trained, and obtains the trained video quality recognition model as the first student model; obtains the first video quality recognition effect of each first student model on the same test data set; screens out K candidate pre-trained models with the highest first video quality recognition effect from the candidate pre-trained model set; uses each candidate pre-trained model in the K candidate pre-trained models and the screened target pre-trained model as a multi-teacher model to train the video quality recognition model to be trained, and obtains the trained video quality recognition model as the second student model; obtains the second video quality recognition effect of each second student model on the same test data set; screens out the candidate pre-trained model with the highest second video quality recognition effect from the K candidate pre-trained models as the target pre-trained model; when the number of the screened target pre-trained models is less than N, deletes the target pre-trained model from the candidate pre-trained model set to obtain an updated candidate pre-trained model set; uses the updated candidate pre-trained model set as the candidate pre-trained model set, and repeats the above process until the number of screened target pre-trained models is N.

[0189] For example, assume that the candidate pre-training model set includes 10 candidate pre-training models, namely M1, M2, M3...M10, K is 4, and N is 3. In the first step, the four candidate pre-training models with the highest first video quality recognition effect are screened out from the candidate pre-training model set (M1, M2, M3, M4, M5, M6, M7, M8, M9, M10), namely M1, M2, M3, and M4; since the target pre-training model has not been screened out in the first step, the candidate pre-training model with the highest first video quality recognition effect is directly screened out from the four candidate pre-training models, such as M1, and M1 is placed in the target pre-training model list. In the second step, the candidate pre-training model set is updated to obtain a new candidate pre-training model set (M2, M3, M4, M5, M6, M7, M8, M9, M10). From the new candidate pre-training model set (M2, M3, M4, M5, M6, M7, M8, M9, M10), the four candidate pre-training models with the highest first video quality recognition effect are screened out, namely M2, M3, M4, and M5; M2, M3, M4, and M5 are respectively combined with the target pre-training model (such as M1) in the target pre-training model list to train the video quality recognition model to be trained, and the second video quality recognition effects of each trained second student model on the same test data set are obtained; from these four candidate pre-training models, the candidate pre-training model with the highest second video quality recognition effect is screened out, such as M2, and M2 is placed in the target pre-training model list. In the third step, the candidate pre-training model set is updated again to obtain a new candidate pre-training model set (M3, M4, M5, M6, M7, M8, M9, M10). From the new candidate pre-training model set (M3, M4, M5, M6, M7, M8, M9, M10), the four candidate pre-training models with the highest first video quality recognition effect are screened out, namely M3, M4, M5, and M6; M3, M4, M5, and M6 are respectively combined with the target pre-training models (such as M1 and M2) in the target pre-training model list to train the video quality recognition model to be trained, and the second video quality recognition effects of each trained second student model on the same test data set are obtained; from these four candidate pre-training models, the candidate pre-training model with the highest second video quality recognition effect is screened out, such as M3, and M3 is put into the target pre-training model list; at this time, the target pre-training model list includes M1, M2 and M3, a total of 3 target pre-training models, and the screening is ended.

[0190] The technical solution provided by the embodiment of the present disclosure is as follows: K candidate pre-training models with the highest first video quality recognition effect are screened out from the candidate pre-training model set; and the candidate pre-training model with the highest second video quality recognition effect is screened out from the K candidate pre-training models as the target pre-training model; the target pre-training model is deleted from the candidate pre-training model set to obtain an updated candidate pre-training model set; the updated candidate pre-training model set is used as the candidate pre-training model set, and the above process is continuously repeated until the number of screened target pre-training models is N. In this way, the model screening method based on the greedy strategy is adopted, which not only enables the screened target pre-training model to better handle the video quality recognition task, but also achieves the best combination effect between the various target pre-training models, thereby making the prediction accuracy of the video quality recognition model subsequently trained based on multiple target pre-training models higher.

[0191] Figure 6 FIG. 1 is a flow chart showing a method for identifying video quality according to an exemplary embodiment. Figure 6 As shown, the video quality identification method is used in a terminal and includes the following steps:

[0192] In step S610, a video to be identified is obtained.

[0193] The video to be identified refers to a video whose video quality needs to be identified, such as a video on the Internet.

[0194] In step S620, the video to be identified is input into the trained video quality identification model to obtain a video quality prediction result of the video to be identified; the trained video quality identification model is obtained by training the video quality identification model to be trained using the sample video, the video quality labeling result of the sample video, the first video quality feature and the first video quality prediction result; the first video quality feature is obtained based on the first target video feature of the sample video outputted by at least two target pre-training models associated with the video quality, and the first video quality prediction result is obtained based on the first video quality feature; the first target video features outputted by different target pre-training models are different.

[0195] The video quality prediction result of the video to be identified refers to the video quality prediction score of the video to be identified, such as 4 points.

[0196] Specifically, the terminal responds to the video quality identification request and obtains the video to be identified; the video to be identified is input into the trained video quality identification model, the trained video quality identification model is used to perform feature extraction processing on the video to be identified, and the extracted video features are used to predict the video quality to obtain the video quality prediction result of the video to be identified.

[0197] For example, the user selects the video to be predicted on the video quality identification interface of the terminal and triggers a video quality identification request. The terminal identifies the video quality prediction score corresponding to the video based on the video quality identification request, and displays the video quality prediction score corresponding to the video on the video quality identification interface for the user to view.

[0198] Furthermore, the training process of the video quality recognition model is as follows: the terminal obtains the sample video and the video quality labeling result of the sample video; the sample video is input into at least two target pre-training models associated with the video quality respectively, and the first target video features of the sample video are outputted correspondingly by the at least two target pre-training models, and the first target video features outputted by the at least two target pre-training models are fused to obtain the first video quality features of the sample video; the first video quality features are classified to obtain the first video quality prediction result of the sample video; the video quality recognition model to be trained is trained according to the sample video, the video quality labeling result of the sample video, the first video quality features and the first video quality prediction result to obtain a trained video quality recognition model.

[0199] In the training method of the above-mentioned video quality recognition model, the trained video quality recognition model outputs the video quality prediction result of the video to be recognized; since the video quality recognition model is trained based on multiple target pre-training models, the first target video features output by each target pre-training model are different, indicating that each target pre-training model focuses on a different type of low-quality video features, so that the trained video quality recognition model can focus on multiple different types of low-quality video features rather than a specific low-quality video feature, thereby improving the recognition accuracy of video quality.

[0200] In an exemplary embodiment, in step S620, the video to be identified is input into the trained video quality recognition model to obtain a video quality prediction result of the video to be identified, which specifically includes the following contents: the video to be identified is input into the trained video quality recognition model to obtain the initial video features of the video to be identified; the initial video features are dimensionally converted to obtain target video features of the video to be identified as video quality features of the video to be identified; the video quality features are classified to obtain video quality classification results of the video to be identified as video quality prediction results.

[0201] The initial video features of the video to be identified refer to the video features output by the trained video quality recognition model, such as Figure 3 Vs.

[0202] The target video features of the video to be identified refer to the video features output by the trained video quality recognition model after dimension conversion, such as Figure 3 The Zs in .

[0203] The dimension conversion process refers to converting the feature dimensions of the initial video features, which can be obtained through full connection processing and normalization processing.

[0204] The video quality feature of the video to be identified refers to feature information related to the video quality of the video to be identified, and can cover various types of low-quality video features.

[0205] Specifically, the terminal inputs the video to be identified into the trained video quality recognition model, performs feature extraction processing on the video to be identified through the trained video quality recognition model, and obtains the initial video features of the video to be identified; inputs the initial video features of the video to be identified into the fully connected layer and the normalization layer, performs dimension conversion processing on the initial video features through the fully connected layer and the normalization layer, and obtains the converted video features of the video to be identified as the target video features of the video to be identified; confirms the target video features of the video to be identified as the video quality features of the video to be identified; inputs the video quality features into the fully connected layer, performs full connection processing on the video quality features through the fully connected layer to classify the video quality features, and obtains the corresponding video quality classification results as the video quality prediction results of the video to be identified.

[0206] For example, refer to Figure 3 Assuming that the video quality recognition model to be trained is a trained video quality recognition model, the terminal inputs the video to be recognized into the trained video quality recognition model, and performs feature extraction processing on the video to be recognized through the trained video quality recognition model to obtain the initial video feature Vs of the video to be recognized; the initial video feature Vs is input into the FC layer (fully connected layer) and the LN layer (normalization layer), and the initial video feature Vs is dimensionally transformed through the FC layer and the LN layer to obtain the target video feature Zs of the video to be recognized; the target video feature Zs of the video to be recognized is confirmed as the video quality feature of the video to be recognized; the video quality feature Zs is input into the fully connected layer, and the video quality feature Zs is fully connected through the fully connected layer to obtain the video quality prediction score of the video to be recognized as the video quality prediction result of the video to be recognized.

[0207] The technical solution provided by the embodiment of the present disclosure inputs the video to be identified into a trained video quality identification model to obtain the initial video features of the video to be identified; performs dimension conversion processing on the initial video features to obtain the target video features of the video to be identified as the video quality features of the video to be identified; performs classification processing on the video quality features to obtain the video quality prediction results of the video to be identified; since the video quality features of the video to be identified can cover multiple types of low-quality video features, it ensures that the quality identification of the video to be identified is more comprehensive, thereby making the obtained video quality prediction results more accurate, and further improving the recognition accuracy of the video quality.

[0208] Figure 7 is a flowchart of another method for training a video quality recognition model according to an exemplary embodiment. Figure 7 As shown, the training method of the video quality recognition model is used in a terminal, comprising the following steps:

[0209] In step S701, a pre-trained model library is constructed; the pre-trained model library includes multiple pre-trained models associated with video quality; the structural differences between the model structures of each pre-trained model meet preset conditions.

[0210] In step S702, for each pre-trained model in the pre-trained model library, if the video quality recognition effect of the student model obtained using the pre-trained model is higher than the video quality recognition effect of the updated model obtained using the pre-trained model, the pre-trained model is used as a candidate pre-trained model; based on the candidate pre-trained models, a candidate pre-trained model set is constructed.

[0211] In step S703, K candidate pre-training models with the highest first video quality recognition effect are screened out from the candidate pre-training model set, and the candidate pre-training model with the highest second video quality recognition effect is screened out from the K candidate pre-training models as the target pre-training model.

[0212] Among them, the first video quality recognition effect is the video quality recognition effect of the first student model obtained by using the candidate pre-training model; the first student model is obtained by training the video quality recognition model to be trained with the candidate pre-training model as the teacher model; the second video quality recognition effect is the video quality recognition effect of the second student model obtained by using the screened candidate pre-training model and the screened target pre-training model; the second student model is obtained by training the video quality recognition model to be trained with the screened candidate pre-training model and the screened target pre-training model as the teacher model; K is a positive integer.

[0213] In step S704, when the number of target pre-trained models is less than N, the target pre-trained model is deleted from the candidate pre-trained model set to obtain an updated candidate pre-trained model set.

[0214] In step S705, the updated candidate pre-training model set is used as the candidate pre-training model set, and the process jumps to step S703 until the number of target pre-training models screened out is N; N is a positive integer; and a pre-training model combination is constructed based on the target pre-training model.

[0215] In step S706, a sample video and a video quality labeling result corresponding to the sample video are obtained.

[0216] In step S707, the sample videos are respectively input into each target pre-training model to obtain each first initial video feature of the sample video; and the first initial video features of the sample video are dimensionally converted to obtain each first target video feature of the sample video.

[0217] In step S708, for each first initial video feature of the sample video, each first initial video feature is input into the gated network to obtain the importance parameter of the target pre-trained model used to output each first initial video feature; the importance parameter of each target pre-trained model is confirmed as the importance parameter of the first target video feature corresponding to the output of each target pre-trained model.

[0218] In step S709, the first target video features are fused according to the importance parameters of the first target video features to obtain the first video quality features of the sample video; the first video quality features are classified to obtain the first video quality classification results of the sample video as the first video quality prediction results.

[0219] In step S710, the sample video is input into the video quality recognition model to be trained to obtain the second initial video feature of the sample video; the second initial video feature of the sample video is dimensionally converted to obtain the second target video feature of the sample video as the second video quality feature of the sample video; the second video quality feature is classified to obtain the second video quality classification result of the sample video as the second video quality prediction result.

[0220] In step S711, a first loss value is obtained based on the difference between the first video quality prediction result and the video quality labeling result, a second loss value is obtained based on the difference between the second video quality prediction result and the video quality labeling result, and a third loss value is obtained based on the difference between the first video quality feature and the second video quality feature; the first loss value, the second loss value and the third loss value are fused to obtain a target loss value.

[0221] In step S712, the video quality recognition model to be trained is trained according to the target loss value until the training end condition is met; the trained video quality recognition model that meets the training end condition is the trained video quality recognition model.

[0222] In the training method of the above-mentioned video quality recognition model, during the model training process, multiple target pre-training models associated with video quality are used to assist in the training of the video quality recognition model. Since the first target video features output by different target pre-training models are different, it means that different target pre-training models focus on different types of low-quality features. Then, the trained video quality recognition model can perceive multiple types of low-quality features, so that the trained video quality recognition model can recognize video quality more comprehensively, thereby improving the accuracy of the output video quality prediction results, which is beneficial to improving the recognition accuracy of video quality; at the same time, the first video quality features, first video quality prediction results and video quality labeling results of the sample videos jointly output by multiple target pre-training models associated with video quality are used as supervision information to assist in the training of the video quality recognition model. Taking into account multiple supervision information, the prediction results of the trained video quality recognition model are more accurate, further improving the recognition accuracy of video quality.

[0223] In order to more clearly illustrate the video quality identification method provided by the embodiment of the present disclosure, the video quality identification method is specifically described below using a specific embodiment. In one embodiment, the present disclosure also provides a video quality assessment method based on a hybrid pre-trained model. By introducing a pre-trained model, the accuracy of the video quality assessment task is improved, overcoming the dependence on a large amount of labeled data; at the same time, a set of pre-trained model screening criteria can be established to achieve good generalization capabilities for different task scenarios. Specifically including the following contents:

[0224] The first step is to build a pre-trained model library.

[0225] In the process of constructing the pre-training model library, we hope to cover as many model types as possible, so that the knowledge contained therein can be aware of various low-quality feature types. Some model selection criteria are defined as follows: (1) The candidate pre-training model needs to achieve good results in its original field, such as a higher score in image classification tasks, higher accuracy in video classification tasks, etc.; good pre-training models are the basis for knowledge transfer. (2) There should be sufficient differences between candidate pre-training models, including model structure, pre-training dataset, parameter quantity, etc.; this ensures that each pre-training model has its own strengths and different pre-training models can complement each other. For example, pre-training models based on video classification can effectively model spatio-temporal relationships, pre-training models based on image classification can perceive content-related features, and pre-training models based on visual text tasks can obtain human emotional expressions; these differences can bring more comprehensive feature representation to the VQA (Video Quality Assessment) task. (3) The pre-training model should be preliminarily verified for effectiveness in the VQA task through KD (Knowledge Distillation).

[0226] Second step, based on the greedy strategy to select the model suitable for video quality identification task.

[0227] Since a single pre-training model cannot guarantee the stability of the prediction results in different task scenarios, the present disclosure selects multiple suitable models for combination when screening pre-training models, and this process mainly adopts a method based on a greedy strategy to select models. First, single pre-training model distillation is performed, and models with better results than simple fine-tuning are combined into a candidate list. Assuming that the candidate list contains H = {h1, h2,..., ht}, the number of models needed is N, and the selected teacher model set S starts from an empty set, and in each step, the best k models are selected from the unselected model list, and then the model effect is evaluated using knowledge distillation, and from the k models, the model with the best distillation result is added to the set S, and this step is repeated N times until the desired number of models N is reached. Wherein, k and N are positive integers, and there is no size relationship between them.

[0228] Third step, use multi-teacher knowledge transfer network structure to train video quality identification model.

[0229] Reference Figure 3 After the model is selected, MKDNet (Multi-teacher Knowledge Distillation Network) is used to train the video quality identification model. As shown in Figure 3As shown, the parameters of the selected pre-trained model are fixed and then used to extract the feature expression of the video. For the image-based pre-trained model, the results of multiple frames are averaged and fused; for the video-based pre-trained model, the corresponding representation can be obtained directly. In order to ensure that the feature dimensions extracted by different pre-trained models can match, the extracted features are transformed through a fully connected layer and mapped to the same dimension. In order to obtain the dynamic weights between each pre-trained model, a gated network is introduced to obtain dynamic responses based on the input features, and the final feature representation is obtained by weighted summation. This step is called MoE (Mixture of Experts, mixed expert model). The network structure of the video quality recognition task then obtains information from the weighted features through knowledge distillation to simulate the features extracted by the pre-trained model. The output prediction result is the video quality. The entire optimization process can be expressed as: L=L1(F s (x),y)+w(L1(F t (x), y) + L2(Zs, Zt)). It should be noted that the meanings of the parameters in the formula refer to the previous related embodiments and are not repeated here.

[0230] refer to Figure 8 After the trained video quality recognition model is obtained, it pays more attention to the low-quality areas in the image, and the video quality prediction results obtained are more accurate. For example, for the same video (No. 3904711682), the MOS (Mean Opinion Score) is 3.04. The video quality prediction score (i.e., Prediction) obtained by the previous method is 3.508, and the video quality prediction score obtained by the trained video quality recognition model is 3.14, which is closer to the MOS, indicating that the prediction results of the trained video quality recognition model are more accurate.

[0231] The above-mentioned video quality assessment method based on hybrid pre-training models can achieve the following technical effects: (1) Use multiple pre-training models related to video quality to extract features, and after fusion, use them as additional supervisory information to assist video quality recognition task training, so that the trained model can pay more attention to different types of low-quality video features; (2) Through the greedy strategy to screen pre-training models suitable for video quality recognition tasks, the constructed model combination shows good generalization ability for different task scenarios and achieves the best effect on different verification sets; (3) Through MKDNet (Multi-teacher Knowledge Distillation Network, multi-teacher knowledge distillation model) to transfer information in the form of knowledge distillation, use feature information to assist video quality recognition tasks during training, and only need to use the video quality recognition model itself in the inference stage, which can significantly reduce computing consumption.

[0232] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0233] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can be referred to each other, and each embodiment focuses on the differences from other embodiments. For related parts, please refer to the description of other method embodiments.

[0234] Based on the same inventive concept, an embodiment of the present disclosure further provides a video quality recognition model training device for implementing the above-mentioned video quality recognition model training method.

[0235] Figure 9 FIG. 1 is a block diagram of a video quality recognition model training device according to an exemplary embodiment. Figure 9 The device includes a sample acquisition unit 910, a sample processing unit 920 and a model training unit 930.

[0236] The sample acquisition unit 910 is configured to acquire a sample video and a video quality labeling result corresponding to the sample video.

[0237] The sample processing unit 920 is configured to input the sample video into the pre-training model combination to obtain the first video quality feature and the first video quality prediction result of the sample video, and input the sample video into the video quality recognition model to be trained to obtain the second video quality feature and the second video quality prediction result of the sample video; the pre-training model combination includes at least two target pre-training models associated with video quality; the first video quality feature is obtained according to the first target video feature of the sample video output corresponding to each target pre-training model, and the first video quality prediction result is obtained according to the first video quality feature; the first target video features output by different target pre-training models are different.

[0238] The model training unit 930 is configured to train the video quality recognition model to be trained based on the difference between the first video quality prediction result and the video quality labeling result, the difference between the second video quality prediction result and the video quality labeling result, and the difference between the first video quality feature and the second video quality feature to obtain a trained video quality recognition model.

[0239] In an exemplary embodiment, the sample processing unit 920 is further configured to input the sample video into each target pre-training model respectively to obtain each first target video feature of the sample video; perform fusion processing on each first target video feature according to the importance parameter of each first target video feature to obtain the first video quality feature of the sample video; perform classification processing on the first video quality feature to obtain the first video quality classification result of the sample video as the first video quality prediction result.

[0240] In an exemplary embodiment, the sample processing unit 920 is further configured to input the sample video into each target pre-training model respectively to obtain each first initial video feature of the sample video; and perform dimensionality conversion processing on each first initial video feature of the sample video to obtain each first target video feature of the sample video.

[0241] In an exemplary embodiment, the training device of the video quality recognition model also includes a parameter determination unit, which is configured to execute for each first initial video feature of the sample video, input each first initial video feature into the gating network, and obtain the importance parameter of the target pre-training model for outputting each first initial video feature; and confirm the importance parameter of each target pre-training model as the importance parameter of the first target video feature corresponding to the output of each target pre-training model.

[0242] In an exemplary embodiment, the sample processing unit 920 is further configured to execute for each target pre-training model, and when the target pre-training model is a video model, the sample video is input into the target pre-training model to obtain the first initial video feature of the sample video; when the target pre-training model is an image model, each sample video frame of the sample video is input into the target pre-training model to obtain the image feature of each sample video frame, and the image features of each sample video frame are fused to obtain the first initial video feature of the sample video.

[0243] In an exemplary embodiment, the sample processing unit 920 is further configured to input the sample video into the video quality recognition model to be trained to obtain a second initial video feature of the sample video; perform dimension conversion processing on the second initial video feature of the sample video to obtain a second target video feature of the sample video as a second video quality feature of the sample video; the dimension of the second target video feature is different from the dimension of the second initial video feature; perform classification processing on the second video quality feature to obtain a second video quality classification result of the sample video as a second video quality prediction result.

[0244] In an exemplary embodiment, the model training unit 930 is further configured to obtain a first loss value based on the difference between the first video quality prediction result and the video quality labeling result, obtain a second loss value based on the difference between the second video quality prediction result and the video quality labeling result, and obtain a third loss value based on the difference between the first video quality feature and the second video quality feature; fuse the first loss value, the second loss value and the third loss value to obtain a target loss value; train the video quality recognition model to be trained according to the target loss value until the training end condition is met; the trained video quality recognition model that meets the training end condition is the trained video quality recognition model.

[0245] In an example embodiment, the training apparatus of the video quality identification model further comprises a model screening unit configured to perform the following steps: constructing a pre-training model library; the pre-training model library comprises a plurality of pre-training models associated with video quality; the structural difference between the model structures of each pre-training model satisfies a preset condition; for each pre-training model in the pre-training model library, if the video quality identification effect of a student model obtained by using the pre-training model is higher than the video quality identification effect of an updated model obtained by using the pre-training model, the pre-training model is taken as a candidate pre-training model; the student model is obtained by training the video quality identification model to be trained by taking the pre-training model as a teacher model, and the updated model is obtained by training the pre-training model; constructing a candidate pre-training model set according to the candidate pre-training models; screening a target pre-training model from the candidate pre-training model set; and constructing the pre-training model according to the target pre-training model.

[0246] In an example embodiment, the model screening unit is further configured to perform the following steps: screening K candidate pre-training models with the highest first video quality identification effects from the candidate pre-training model set, and screening a candidate pre-training model with the highest second video quality identification effect from the K candidate pre-training models as the target pre-training model; the first video quality identification effect is the video quality identification effect of a first student model obtained by using the candidate pre-training model; the first student model is obtained by training the video quality identification model to be trained by taking the candidate pre-training model as a teacher model; the second video quality identification effect is the video quality identification effect of a second student model obtained by using the screened candidate pre-training model and the screened target pre-training model; the second student model is obtained by training the video quality identification model to be trained by taking the screened candidate pre-training model and the screened target pre-training model as teacher models; and K is a positive integer.

[0247] In an example embodiment, the model screening unit is further configured to perform the following steps: when the number of target pre-training models is less than N, deleting the target pre-training models from the candidate pre-training model set to obtain an updated candidate pre-training model set; taking the updated candidate pre-training model set as the candidate pre-training model set, and jumping to the steps of screening K candidate pre-training models with the highest first video quality identification effects from the candidate pre-training model set, and screening a candidate pre-training model with the highest second video quality identification effect from the K candidate pre-training models as the target pre-training model, until the number of screened target pre-training models is N; and N is a positive integer.

[0248] Based on the same inventive concept, the embodiments of the present disclosure further provide a video quality identification apparatus for implementing the video quality identification method described above.

[0249] Figure 10 is a block diagram of a video quality identification device according to an example embodiment. Referring to Figure 10 The device includes a video acquisition unit 1010 and a quality identification unit 1020.

[0250] The video acquisition unit 1010 is configured to perform acquisition of a video to be identified.

[0251] The quality identification unit 1020 is configured to perform inputting of the video to be identified into a trained video quality identification model to obtain a video quality prediction result of the video to be identified; the trained video quality identification model is obtained by training a video quality identification model to be trained by using a sample video, a video quality label result of the sample video, a first video quality feature, and a first video quality prediction result; the first video quality feature is obtained according to first target video features of the sample video output by at least two target pre-training models associated with video quality, and the first video quality prediction result is obtained according to the first video quality feature; the first target video features output by different target pre-training models are different.

[0252] In an example embodiment, the quality identification unit 1020 is further configured to perform inputting of the video to be identified into the trained video quality identification model to obtain an initial video feature of the video to be identified; performing dimension conversion processing on the initial video feature to obtain a target video feature of the video to be identified as a video quality feature of the video to be identified; performing classification processing on the video quality feature to obtain a video quality classification result of the video to be identified as the video quality prediction result.

[0253] As to the device in the above-described embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0254] The various modules in the above-described video quality identification model training device or video quality identification device can be all or partially implemented by software, hardware, and combinations thereof. The various modules described above can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform operations corresponding to the various modules.

[0255] Figure 11 is a block diagram of an electronic device 1100 for implementing a video quality identification model training method or a video quality identification method according to an example embodiment. For example, the electronic device 1100 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0256] Referring to Figure 11 , the electronic device 1100 may include one or more of the following components: a processing component 1102 , a memory 1104 , a power component 1106 , a multimedia component 1108 , an audio component 1110 , an input / output (I / O) interface 1112 , a sensor component 1114 , and a communication component 1116 .

[0257] The processing component 1102 generally controls the overall operation of the electronic device 1100, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 1102 may include one or more processors 1120 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 1102 may include one or more modules to facilitate interaction between the processing component 1102 and other components. For example, the processing component 1102 may include a multimedia module to facilitate interaction between the multimedia component 1108 and the processing component 1102.

[0258] The memory 1104 is configured to store various types of data to support operations on the electronic device 1100. Examples of such data include instructions for any application or method operating on the electronic device 1100, contact data, phone book data, messages, pictures, videos, etc. The memory 1104 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, optical disk, or graphene memory.

[0259] The power supply component 1106 provides power to the various components of the electronic device 1100. The power supply component 1106 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1100.

[0260] The multimedia component 1108 includes a screen to provide an output interface between the electronic device 1100 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect duration and intensity of the touching or sliding action. In some embodiments, the multimedia component 1108 includes a front camera and / or a rear camera. The front camera and / or the rear camera can receive external multimedia data when the electronic device 1100 is in an operating mode, such as a shooting mode or a video mode. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom.

[0261] The audio component 1110 is configured to output and / or input an audio signal. For example, the audio component 1110 includes a microphone (MIC) configured to receive an external audio signal when the electronic device 1100 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 1104 or transmitted via the communication component 1116. In some embodiments, the audio component 1110 also includes a speaker to output an audio signal.

[0262] The I / O interface 1112 provides an interface between the processing component 1102 and peripheral interface modules, which can be a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0263] The sensor component 1114 includes one or more sensors to provide various state assessments for the electronic device 1100. For example, the sensor component 1114 can detect an open / closed state of the electronic device 1100, relative positioning of components, such as a display and a keypad of the electronic device 1100, a change in position of the electronic device 1100 or components of the electronic device 1100, presence or absence of user contact with the electronic device 1100, a direction or acceleration / deceleration of the device 1100, and a change in temperature of the electronic device 1100. The sensor component 1114 can include a proximity sensor configured to detect presence of a nearby object without any physical touch. The sensor component 1114 can further include a light sensor such as a CMOS or CCD image sensor for use in an imaging application. In some embodiments, the sensor component 1114 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0264] The communication component 1116 is configured to facilitate wired or wireless communication between the electronic device 1100 and other devices. The electronic device 1100 can access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 1116 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1116 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0265] In an exemplary embodiment, the electronic device 1100 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described methods.

[0266] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1104 including instructions, and the instructions can be executed by the processor 1120 of the electronic device 1100 to perform the above method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0267] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions, and the instructions can be executed by the processor 1120 of the electronic device 1100 to implement the above method.

[0268] It should be noted that the above-mentioned devices, electronic devices, computer-readable storage media, computer program products, etc. can also include other implementation methods according to the description of the method embodiments. The specific implementation methods can refer to the description of the relevant method embodiments and will not be described one by one here.

[0269] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

[0270] It should be understood that the present disclosure is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the present disclosure. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training a video quality recognition model, characterized in that: include: Obtaining a sample video and a video quality labeling result corresponding to the sample video; Inputting the sample video into a pre-trained model combination to obtain a first video quality feature and a first video quality prediction result of the sample video, and inputting the sample video into a video quality recognition model to be trained to obtain a second video quality feature and a second video quality prediction result of the sample video; the pre-trained model combination includes at least two target pre-trained models associated with video quality, the first video quality feature is obtained based on the first target video feature of the sample video outputted by each target pre-trained model, and the first video quality prediction result is obtained based on the first video quality feature; The first target video features output by different target pre-training models are different; According to the difference between the first video quality prediction result and the video quality labeling result, the difference between the second video quality prediction result and the video quality labeling result, and the difference between the first video quality feature and the second video quality feature, the video quality recognition model to be trained is trained to obtain a trained video quality recognition model.

2. The method according to claim 1, characterized in that The step of inputting the sample video into a target pre-trained model combination to obtain a first video quality feature and a first video quality prediction result of the sample video includes: Inputting the sample videos into respective target pre-training models to obtain respective first target video features of the sample videos; performing fusion processing on each of the first target video features according to the importance parameters of each of the first target video features to obtain a first video quality feature of the sample video; Classification processing is performed on the first video quality feature to obtain a first video quality classification result of the sample video as the first video quality prediction result.

3. The method according to claim 2, characterized in that The step of inputting the sample videos into respective target pre-training models to obtain respective first target video features of the sample videos comprises: Inputting the sample videos into respective target pre-training models to obtain respective first initial video features of the sample videos; Dimension conversion processing is performed on each first initial video feature of the sample video to obtain each first target video feature of the sample video.

4. The method according to claim 3, characterized in that Before fusing the first target video features according to the importance parameters of the first target video features to obtain the first video quality feature of the sample video, the method further includes: For each first initial video feature of the sample video, input each first initial video feature into a gating network to obtain an importance parameter of a target pre-trained model for outputting each first initial video feature; The importance parameter of each target pre-training model is confirmed as the importance parameter of the first target video feature outputted by each target pre-training model.

5. The method according to claim 3, characterized in that The step of inputting the sample videos into respective target pre-training models to obtain respective first initial video features of the sample videos includes: For each target pre-training model, if the target pre-training model is a video model, input the sample video into the target pre-training model to obtain a first initial video feature of the sample video; In the case that the target pre-trained model is an image model, each sample video frame of the sample video is input into the target pre-trained model to obtain image features of each sample video frame, and the image features of each sample video frame are fused to obtain the first initial video features of the sample video.

6. The method according to claim 1, characterized in that Inputting the sample video into the video quality recognition model to be trained to obtain the second video quality feature and the second video quality prediction result of the sample video includes: Inputting the sample video into a video quality recognition model to be trained to obtain a second initial video feature of the sample video; Performing a dimensionality conversion process on the second initial video feature of the sample video to obtain a second target video feature of the sample video as a second video quality feature of the sample video; the dimension of the second target video feature is different from the dimension of the second initial video feature; The second video quality feature is classified to obtain a second video quality classification result of the sample video as the second video quality prediction result.

7. The method according to claim 1, characterized in that The method of training the video quality recognition model to be trained based on the difference between the first video quality prediction result and the video quality labeling result, the difference between the second video quality prediction result and the video quality labeling result, and the difference between the first video quality feature and the second video quality feature to obtain a trained video quality recognition model includes: Obtaining a first loss value based on a difference between the first video quality prediction result and the video quality labeling result, obtaining a second loss value based on a difference between the second video quality prediction result and the video quality labeling result, and obtaining a third loss value based on a difference between the first video quality feature and the second video quality feature; fusing the first loss value, the second loss value, and the third loss value to obtain a target loss value; According to the target loss value, the video quality recognition model to be trained is trained until a training end condition is reached; the trained video quality recognition model that reaches the training end condition is the trained video quality recognition model.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: Constructing a pre-trained model library; the pre-trained model library includes a plurality of pre-trained models associated with video quality; the structural differences between the model structures of each pre-trained model meet preset conditions; For each pre-trained model in the pre-trained model library, if the video quality recognition effect of the student model obtained by using the pre-trained model is higher than the video quality recognition effect of the updated model obtained by using the pre-trained model, the pre-trained model is used as a candidate pre-trained model; wherein the student model is obtained by training the video quality recognition model to be trained using the pre-trained model as a teacher model, and the updated model is obtained by training the pre-trained model; Constructing a candidate pre-training model set based on the candidate pre-training models; Filtering the target pre-training model from the candidate pre-training model set; Construct the pre-training model combination according to the target pre-training model.

9. The method according to claim 8, characterized in that The step of selecting the target pre-trained model from the candidate pre-trained model set includes: Selecting K candidate pre-training models with the highest first video quality recognition effect from the candidate pre-training model set, and selecting a candidate pre-training model with the highest second video quality recognition effect from the K candidate pre-training models as the target pre-training model; Among them, the first video quality recognition effect is the video quality recognition effect of the first student model obtained by using the candidate pre-training model; the first student model is obtained by training the video quality recognition model to be trained with the candidate pre-training model as the teacher model; the second video quality recognition effect is the video quality recognition effect of the second student model obtained by using the screened candidate pre-training model and the screened target pre-training model; the second student model is obtained by training the video quality recognition model to be trained with the screened candidate pre-training model and the screened target pre-training model as the teacher model; K is a positive integer.

10. The method according to claim 9, characterized in that Also includes: When the number of the target pre-trained models is less than N, deleting the target pre-trained model from the candidate pre-trained model set to obtain an updated candidate pre-trained model set; The updated candidate pre-training model set is used as the candidate pre-training model set, and the step of screening out K candidate pre-training models with the highest first video quality recognition effect from the candidate pre-training model set, and screening out the candidate pre-training model with the highest second video quality recognition effect from the K candidate pre-training models as the target pre-training model is performed until the number of screened target pre-training models is N, where N is a positive integer.

11. A video quality identification method, characterized in that: include: Get the video to be identified; Inputting the video to be identified into a trained video quality identification model to obtain a video quality prediction result of the video to be identified; The trained video quality recognition model is obtained by training the video quality recognition model to be trained using a sample video, a video quality labeling result of the sample video, a first video quality feature, and a first video quality prediction result; the first video quality feature is obtained based on the first target video features of the sample video outputted by at least two target pre-training models associated with video quality, and the first video quality prediction result is obtained based on the first video quality feature; The first target video features output by different target pre-training models are different.

12. The method according to claim 11, characterized in that The step of inputting the video to be identified into a trained video quality identification model to obtain a video quality prediction result of the video to be identified includes: Inputting the video to be identified into a trained video quality recognition model to obtain initial video features of the video to be identified; Performing dimension conversion on the initial video features to obtain target video features of the video to be identified as video quality features of the video to be identified; The video quality features are classified to obtain a video quality classification result of the video to be identified as the video quality prediction result.

13. A training device for a video quality recognition model, characterized in that: include: A sample acquisition unit is configured to acquire a sample video and a video quality labeling result corresponding to the sample video; The sample processing unit is configured to input the sample video into a pre-trained model combination to obtain a first video quality feature and a first video quality prediction result of the sample video, and input the sample video into a video quality recognition model to be trained to obtain a second video quality feature and a second video quality prediction result of the sample video; the pre-trained model combination includes at least two target pre-trained models associated with video quality; the first video quality feature is obtained based on the first target video feature of the sample video outputted by each target pre-trained model, and the first video quality prediction result is obtained based on the first video quality feature; The first target video features output by different target pre-training models are different; The model training unit is configured to train the video quality recognition model to be trained based on the difference between the first video quality prediction result and the video quality labeling result, the difference between the second video quality prediction result and the video quality labeling result, and the difference between the first video quality feature and the second video quality feature to obtain a trained video quality recognition model.

14. A video quality identification device, characterized in that: include: A video acquisition unit is configured to acquire a video to be identified; The quality recognition unit is configured to execute inputting the video to be recognized into a trained video quality recognition model to obtain a video quality prediction result of the video to be recognized; the trained video quality recognition model is obtained by training the video quality recognition model to be trained using a sample video, a video quality labeling result of the sample video, a first video quality feature, and a first video quality prediction result; the first video quality feature is obtained based on the first target video feature of the sample video outputted by at least two target pre-trained models associated with video quality, and the first video quality prediction result is obtained based on the first video quality feature; The first target video features output by different target pre-training models are different.

15. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the video quality identification model training method according to any one of claims 1 to 10, or the video quality identification method according to any one of claims 11 to 12.

16. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the video quality identification model training method according to any one of claims 1 to 10, or the video quality identification method according to any one of claims 11 to 12.

17. A computer program product comprising instructions, characterized in that: When the instruction is executed by a processor of an electronic device, the electronic device is enabled to execute the video quality identification model training method according to any one of claims 1 to 10, or the video quality identification method according to any one of claims 11 to 12.

Images