Methods, apparatus, equipment and storage media for determining video quality

Abstract

This disclosure relates to a method, apparatus, device, and storage medium for determining video quality. The video quality determination method includes: acquiring a target video and related text information of the target video; the related text information is used to describe the video content of the target video. The preprocessed target video and related text information are input into a trained video classification model. After classification processing by the video classification model, the confidence score of each video quality category in the video classification model is output. The confidence score represents the probability that the target video belongs to the corresponding video quality category. Based on the confidence score of each video quality category corresponding to the target video, the quality determination result of the target video is obtained. This disclosure uses information from two different modalities—video and text—and a video classification model to predict the quality of a target video. Since information from both video and text modalities can comprehensively characterize the target video, this disclosure can more accurately determine the quality of the target video.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for determining video quality. Background Technology

[0002] The short video industry is developing rapidly, but with the increase in the number of short videos, the quality of short videos varies greatly. In order to ensure the user's browsing experience of video content, the quality of short videos needs to be reviewed in advance before being pushed to users.

[0003] Therefore, determining video quality is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure provides a method, apparatus, device, and storage medium for determining video quality, which can accurately determine video quality by utilizing multimodal information of the video.

[0005] In a first aspect, embodiments of this disclosure provide a method for determining video quality, including:

[0006] Obtain the target video and related text information of the target video; wherein, the related text information is used to describe the video content of the target video;

[0007] The preprocessed target video and related text information are input into a trained video classification model. After classification by the video classification model, the confidence score of each video quality category in the video classification model is output; wherein, the confidence score is used to represent the probability that the target video belongs to the corresponding video quality category.

[0008] Based on the confidence level of each video quality category corresponding to the target video, the quality determination result of the target video is obtained.

[0009] Secondly, embodiments of this disclosure also provide a video quality determination apparatus, comprising:

[0010] The first acquisition unit is used to acquire a target video and related text information of the target video; wherein, the related text information is used to describe the video content of the target video;

[0011] The model processing unit is used to input the preprocessed target video and the related text information into a trained video classification model, and after classification processing by the video classification model, output the confidence score of each video quality category in the video classification model; wherein, the confidence score is used to represent the probability that the target video belongs to the corresponding video quality category;

[0012] The second acquisition unit is used to acquire the quality determination result of the target video based on the confidence level of each video quality category corresponding to the target video.

[0013] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0014] Memory;

[0015] Processor; and

[0016] Computer programs;

[0017] The computer program is stored in the memory and configured to be executed by the processor to implement the video quality determination method described above.

[0018] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the video quality determination method described above.

[0019] This disclosure provides a method for determining video quality. The method involves acquiring a target video and related text information, inputting the target video and related text information into a data preprocessing module, and then inputting this data into a pre-trained video classification model for quality category prediction. This yields the confidence score for each quality category corresponding to the target video. Based on the confidence score for each quality category, the method obtains the quality determination result for the target video. This disclosure uses information from both video and text modalities to predict the quality of the target video using a video classification model. Since information from both video and text modalities can comprehensively characterize the target video, this disclosure can more accurately determine the quality of the target video, thereby providing a more accurate assessment of the determined video quality. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A schematic diagram illustrating an application scenario provided by an embodiment of this disclosure;

[0023] Figure 2A flowchart illustrating a training method for a video classification model provided in this embodiment of the present disclosure;

[0024] Figure 3 A schematic diagram of a first training phase provided in an embodiment of this disclosure;

[0025] Figure 4 This is a schematic diagram of the structure of a second-stage model provided in an embodiment of the present disclosure;

[0026] Figure 5 A flowchart illustrating a video quality determination method provided in an embodiment of this disclosure;

[0027] Figure 6 This is a schematic diagram of the structure of a video quality determination device provided in an embodiment of the present disclosure;

[0028] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0029] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0030] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0031] Currently, artificial intelligence technology has been widely applied in the short video industry. For example, in the scenario of quality scoring for short videos in the automotive vertical category, most models are based on video frame information using manually labeled data. The labels of the manually labeled data are mostly discrete binary quality category labels such as "high quality" and "low quality". The binary classification model is mostly based on video frame information, and the confidence level of the "high quality" category is used as the quality score of the short video in the automotive vertical category. However, the current modeling method only uses video frame information. At the same time, the binary classification modeling method has the problem of low label granularity, which leads to the problem of low accuracy in predicting quality scores.

[0032] To address the aforementioned technical problems, this disclosure provides a video quality determination method. First, it acquires two different modalities of information: the target video and related textual information. Then, using a video classification model, it performs a more comprehensive and accurate quality category prediction based on the multimodal information of the target video, obtaining the confidence level of the target video for each quality category. Based on the confidence level of each quality category, it obtains the quality determination result of the target video. Because this disclosure can predict the video quality based on the multimodal information of the target video, the obtained quality determination result can more accurately reflect the quality of the target video, improving the accuracy of video quality determination.

[0033] Furthermore, to more accurately predict the quality of the target video, this embodiment sets multiple video quality categories for the video classification model to increase the granularity of the labels. The confidence scores of each video quality category are then fused and calculated to obtain a quality score that more accurately represents the quality of the target video. This effectively improves the accuracy of the quality prediction. Based on this, it can also be used in conjunction with a video recommendation system to recommend videos based on the determined quality score of the target video, thereby improving the quality of the videos recommended to users. Detailed descriptions are provided below through one or more embodiments.

[0034] Specifically, the video quality determination method can be executed by either the terminal or the server. Specifically, the terminal or server can predict the quality of the target video based on a video classification model. The entity executing the training method for the video classification model and the entity executing the quality determination method can be the same or different.

[0035] For example, in one application scenario, such as Figure 1 As shown, server 12 trains a video classification model. Terminal 11 obtains the trained video classification model from server 12, and uses this model to predict the quality of the target video. The target video can be captured by terminal 11. Alternatively, the target video can be obtained by terminal 11 from another device. Or, the target video can be obtained by terminal 11 after editing a preset video, which can be captured by terminal 11 or obtained by terminal 11 from another device. Here, no specific limitation is made to the other devices.

[0036] In another application scenario, server 12 trains a video classification model. Furthermore, server 12 predicts the quality of the target video based on the trained video classification model. The method by which server 12 acquires the target video can be similar to the method by which terminal 11 acquires the target video as described above, and will not be repeated here.

[0037] In another application scenario, terminal 11 trains a video classification model. Furthermore, terminal 11 predicts the quality of the target video based on the trained video classification model.

[0038] It is understood that the training method for the video classification model and the video quality determination method provided in this disclosure are not limited to the possible scenarios described above. Since the trained video classification model is applied in the video quality determination method, the training method for the video classification model can be introduced first before introducing the video quality determination method.

[0039] The following section uses the training method of a video classification model applied to server 12 as an example to introduce a training method for a video classification model, namely, the training process of a video classification model. It is understood that this training method is also applicable to the scenario of training a video classification model on terminal 11. The type of training samples for the video classification model can be determined according to user needs and is not limited here.

[0040] Figure 2 This is a flowchart illustrating a training method for a video classification model provided in an embodiment of the present disclosure, specifically including the following: Figure 2 Steps S210 to S230 are shown below:

[0041] Understandably, the training of a video classification model is divided into two stages. The first training stage is based on a large number of unlabeled video samples (e.g., short video samples in the automotive category) and text samples for multimodal self-supervised training. The second training stage is based on the model obtained in the first training stage, and then trained on labeled video samples (again, short video samples in the automotive category) and text samples. Specifically, the model part in the first training stage is referred to as the first-stage model, and the model part in the second training stage is referred to as the second-stage model.

[0042] S210. Obtain the first training sample set and the second training sample set;

[0043] The first training sample set includes a first video training sample and a first text training sample with a corresponding relationship. The first text training sample is used to describe the video content of the corresponding first video training sample. The second training sample set includes a second video training sample, a second text training sample and a video quality category label with a corresponding relationship. The second text training sample is used to describe the video content of the corresponding second video training sample.

[0044] In practical applications, the second training sample set includes training samples with video quality category labels, which are pre-labeled labels used to indicate the true quality category to which the training samples belong.

[0045] Optionally, the first training sample set includes samples corresponding to multiple preset learning tasks. The first training sample set includes first video training samples and first text training samples. The relevant text information of the first video training samples, such as title information and other text information that can describe the video content of the first video training samples, can be used as the first text training samples.

[0046] Understandably, the first training sample set includes a large number of unlabeled first video training samples and first text training samples, used to train the first-stage model. Unlabeled samples are easy to obtain, numerous, and require no labeling costs. The readily available and large number of training samples enhance the multimodal feature extraction capabilities of the feature processing module obtained in the first training stage, and are highly feasible. The first training sample set is used for training multiple pre-defined learning tasks, which include at least video learning tasks and text learning tasks. The first training sample set includes first video training samples corresponding to the video learning tasks and first text training samples corresponding to the text learning tasks. The second training sample set includes labeled second video training samples and second text training samples, used to train the second-stage model to obtain a video classification model. Specifically, it includes second video training samples and second text training samples that correspond to the second video training samples. The second video training samples and second text training samples serve as inputs to the model. Each second video training sample and each second text training sample in the second training sample set also has a labeled true quality category, which is the label corresponding to the sample. Supervised training of the video classification model can be performed based on the input samples and their corresponding labels.

[0047] S220. The first-stage model is self-supervised using the first training sample set to obtain the trained first-stage model.

[0048] The modules participating in the first training phase include a data preprocessing module and a first-stage model. The first-stage model includes a feature processing module and learning modules corresponding to each preset learning task. The first data preprocessing module and the feature processing module corresponding to the first-stage model are used to generate real sample labels for each preset learning task. The feature processing module is used to generate sample fusion features based on the preprocessing results of the first data preprocessing module. The learning module is used to output the prediction results for the corresponding preset learning task. The prediction results of each preset learning task and the real sample labels of each preset learning task are used to update the model parameters of the first-stage model.

[0049] Understandably, based on the above S210, the first training phase is a self-supervised learning process for multiple tasks based on multimodal information. Each learning module is used for a pre-defined learning task, and multiple learning modules can include a first learning module and a second learning module. The first learning module is used for video learning tasks, and the second learning module is used for text learning tasks. The trained first-stage model is obtained by completing each pre-defined learning task.

[0050] For example, see Figure 3 , Figure 3 This is a schematic diagram of a first training phase provided in an embodiment of this disclosure. Figure 3The first training phase includes a first data preprocessing module and a first-stage model. The first-stage model includes a feature processing module and multiple learning modules. The feature processing module includes a sequence feature acquisition module and a feature fusion module. The first data preprocessing module includes multiple preprocessing sub-modules, one for each modality (text modality and video modality). These sub-modules include at least a first preprocessing sub-module and a second preprocessing sub-module. The first preprocessing sub-module processes video modalities, and the second preprocessing sub-module processes text modalities. The first preprocessing sub-module can process video training samples using frame extraction, while the second preprocessing sub-module can process text training samples using the data preprocessing methods found in the Chinese pre-trained model (RoBERTa). The sequence feature acquisition module includes multiple acquisition sub-modules, one for each preprocessing sub-module. These sub-modules include a first acquisition sub-module and a second acquisition sub-module. The first acquisition sub-module processes video modalities, and the second acquisition sub-module processes text modalities. The first acquisition sub-module can use an image classification model (Vision). The Transformer (ViT) module extracts sequence features from the video modality. The second acquisition submodule can use the token ID feature mapping method in RoBERTa to obtain sequence features from the text modality. The sequence features output from multiple acquisition submodules are input to the feature fusion module for fusion processing. The feature fusion module includes a feature concatenation module and a multimodal feature interaction fusion module. The multimodal feature interaction fusion module can use the Encoder module in RoBERTa. The sequence features output from multiple acquisition submodules are processed sequentially by the feature concatenation module and the multimodal feature interaction fusion module to obtain the first sample fusion features. Based on the first sample fusion features, the input features of the learning module corresponding to each preset learning task are determined. The learning module generates sample prediction results for each preset learning task based on the input features. The model parameters of the first-stage model are updated based on the sample prediction results and the real sample labels of each preset learning task. The first acquisition submodule may not update its model parameters during training, while the model parameters of the second acquisition submodule and the feature fusion module will be updated during training. The specific model structure of the learning modules corresponding to different preset learning tasks is not limited here and can be determined according to user needs.

[0051] Optionally, training the first-stage model in S220 above can be achieved through the following steps:

[0052] The training samples from the first training sample set are input into the first data preprocessing module and then into the feature processing module to generate the first sample fusion features and the real sample labels for each preset learning task. Based on the first sample fusion features, the input features of the learning module corresponding to each preset learning task are determined. The learning module generates the sample prediction results for each preset learning task based on the input features. The model parameters of the first stage model are updated based on the sample prediction results and the real sample labels for each preset learning task.

[0053] Understandably, after the training samples from the first training sample set are input into the first data preprocessing module, preprocessing results are obtained. These preprocessing results are then input into the feature processing module for sequence feature acquisition and feature fusion, resulting in the first sample fusion features output by the feature fusion module and the real sample labels for each preset learning task output by the first data preprocessing module and the sequence feature acquisition module. Subsequently, based on the first sample fusion features, the input features for the learning modules corresponding to each preset learning task are determined. Different preset learning tasks require different input features; for example, the first input features required for training the first learning module and the second input features required for training the second learning module are determined. For any preset learning task, after determining the input features, these features are used as the input to the learning module. After passing through the corresponding learning module, the sample prediction results corresponding to the preset learning task are output, and the loss value is calculated based on the sample prediction results and the real sample labels corresponding to the preset learning task. The model parameters of the first-stage model are updated according to the loss values ​​of all preset learning tasks. After the model convergence condition is met, the self-supervised training of the first-stage model is completed.

[0054] Optionally, the real sample labels for each preset learning task include real video sample features and real text sample label sequences.

[0055] Optionally, the generation of the first sample fusion features and the real sample labels for each preset learning task can be achieved through the following steps:

[0056] Features are extracted from the first video training sample to obtain video real sample features. The first text training sample is serialized and encoded to obtain a text real sample label sequence. The features of the position to be predicted in the video real sample features are masked to generate video mask sample features. The text labels of the position to be predicted in the text real sample label sequence are masked to generate a text mask sample sequence, and the text mask sample sequence is feature mapped to obtain text mask sample features. The video mask sample features and the text mask sample features are concatenated according to a set concatenation rule to obtain concatenated mask sample features. The concatenated mask sample features are subjected to multimodal feature interaction fusion processing to obtain the first sample fusion feature.

[0057] Understandably, data preprocessing for different modalities can be performed in parallel or sequentially, with no specific order limiting the processing. Specifically, frames are extracted from the first video training sample at a preset frequency to obtain a video frame sample sequence, for example, at a preset frequency of one frame per second. After extracting frames from the video training sample, feature extraction is performed on the video frame sample sequence to obtain multi-dimensional video modal sequence features. It is understood that the sequence length of the corresponding video mask sample features in the spliced ​​mask sample features is fixed. When extracting features from the video frame sample sequence, if the length of the target video frame sequence obtained by extracting frames from the target video sample based on the preset frequency is less than the fixed sequence length of the video mask sample features, then during the feature extraction process of the video frame sample sequence, the target video... The sequence features are padded with all zeros based on the frame sequence length until the length of the video modal sequence features reaches the fixed sequence length of the video mask sample features. The sequence positions corresponding to the all-zero sequence features are ignored during model calculation. If the length of the target video frame sequence is greater than the fixed sequence length of the video mask sample features, the target video frame sequence is sampled at equal intervals until the number of sampled frames is equal to the fixed sequence length of the video mask sample features. For example, if the fixed sequence length of the video mask sample features is 10, extracting video modal sequence features from one video training sample yields 10*768-dimensional video modal sequence features, and extracting video modal sequence features from a batch of video samples (a batch of video samples includes 100 video samples) yields 100*10*768-dimensional video modal sequence features. After completing the video modal sequence feature extraction, the video modal sequence features are randomly masked in the sequence dimension to generate video mask sample features. Subsequently, the entire video modal sequence features can be directly used as the video real sample features, or the features of the sequence positions to be predicted in the video modal sequence features can be used as the video real sample features. Meanwhile, when preprocessing a single title sample, for example, if the fixed input sequence length of the text modality is 10, the title sample is serialized and encoded using the data preprocessing method in RoBERTa to obtain a text sample label sequence with a sequence length of 10, denoted as the token id sequence. The text sample label sequence is then randomly masked using the masking method of the pre-training method (Masked Language Model, MLM) in RoBERTa to generate a text mask sample sequence. The text mask sample sequence is then converted into text mask sample features using the feature mapping method of the token id in RoBERTa. Here, the entire text sample label sequence can be directly used as the text real sample label sequence, or the text label at the position of the sequence to be predicted in the text sample label sequence can be used as the text real sample label sequence. The following embodiment uses the entire text sample label sequence as the text real sample label sequence and the entire video modality sequence features as the video real sample features as examples for illustration.Understandably, the last dimension of the video mask sample features and the text mask sample features are the same, and the preset frequency can be determined by the user according to their needs. After masking, the video mask sample features and the text mask sample features are concatenated to obtain concatenated mask sample features. These concatenated mask sample features are then input into the Encoder module for multimodal feature interaction and fusion processing to obtain the first sample fusion feature. The fusion feature is then segmented and used as input to each learning module. The loss between the video modal sample prediction result predicted by the first learning module and the real video sample features is calculated using a contrastive learning approach to obtain the first loss value. The loss between the text modal sample prediction result predicted by the second learning module and the real text sample labeled sequence is calculated using a multi-classification task approach to obtain the second loss value. The text modal sequence positions corresponding to unmasked token IDs are ignored when calculating the loss value. The model parameters of the first-stage model are then updated based on the first and second loss values. Understandably, in the first stage of multimodal self-supervised learning based on unlabeled samples, training the first-stage model through multiple learning tasks can effectively improve the multimodal feature extraction capability of the feature processing module.

[0058] Optionally, the above-mentioned determination of the input features of the learning module corresponding to each preset learning task based on the fusion features of the first sample can be achieved through the following steps:

[0059] When the preset learning task includes a video learning task, the learning module corresponding to the preset learning task is used to obtain the sequence position of the video mask sample features in the spliced ​​mask sample features, and the input features of the learning module corresponding to the video learning task are determined in the first sample fusion features based on the sequence position.

[0060] When the preset learning task includes a text learning task, the first sample fusion feature is determined as the input feature of the learning module corresponding to the text learning task.

[0061] Understandably, for a video training sample, for example, a video mask sample feature with a fixed sequence length of 10, after preprocessing, feature extraction, and masking, the video training sample yields a video mask sample feature with a sequence length of 10 and a dimension of 10*768. For a text training sample, for example, a text modality with a fixed input sequence length of 10, the title sample, after preprocessing, yields 10 token IDs. After masking, the 10 token IDs are converted into 10*768-dimensional text mask sample features using the token ID feature mapping method in RoBERTa. The video mask sample features and text mask sample features are concatenated according to a set concatenation rule to obtain concatenated mask sample features. Based on the sequence position of the video mask sample features in the concatenated mask sample features, the sample fusion feature is divided into two parts. The first part of the sample features serves as the input feature of the first learning module corresponding to the video learning task, and the entire sample fusion feature serves as the input feature of the second learning module corresponding to the text learning task.

[0062] S230. Construct a second-stage model based on the quality classification module and the feature processing module in the trained first-stage model, and train the second-stage model using the second training sample set to obtain the trained second-stage model, which serves as the trained video classification model.

[0063] The feature processing module updates parameters during the second training phase.

[0064] Optionally, the video classification model trained in S230 above is achieved through the following steps:

[0065] The second training phase utilizes a second data preprocessing module and a feature processing module to perform data preprocessing, sequence feature acquisition, and feature fusion on the training samples in the second training sample set (including corresponding second video training samples and second text training samples) to obtain second sample fusion features. These second sample fusion features are then input into the quality classification module in the second-stage model. After processing by the quality classification module, the classification prediction result for the second video training samples is output. A loss value is calculated based on the classification prediction result and the true quality category of the second video training samples. The model parameters of the second-stage model are updated according to the loss value until the model convergence condition is met, thus obtaining the video classification model.

[0066] Understandably, based on S210 and S220 above, a video classification model is constructed based on the quality classification module and the feature processing module of the trained first-stage model. That is, the feature processing module used in the second training stage comes from the feature processing module of the model trained in the first training stage. During training and inference, neither the video modality nor the text modality requires masking. The initial model parameters corresponding to the feature processing module during the second training stage come from the feature processing module of the model trained in the first training stage. The quality classification module is then connected after the feature processing module, essentially replacing multiple learning modules in the first-stage model for supervised training. After the video classification model is constructed, the second training samples are input into the second data preprocessing module and then into the video classification model. The feature processing module extracts the fused features of the second samples, which are then input into the quality classification module for quality category prediction, outputting the predicted classification result. In the model training process, the first training stage performs self-supervised training, and the second training stage performs supervised training of the video classification model based on the feature processing module trained in the first training stage, which can improve the accuracy of the video classification model. Subsequently, the loss value is calculated based on the predicted classification results and the labeled true quality categories. The model parameters of the second-stage model are then updated using the loss value until the model convergence condition is met, resulting in a trained video quality classification model.

[0067] For example, see Figure 4 , Figure 4 This is a schematic diagram of the structure of a second training model provided in an embodiment of this disclosure. Figure 4 The second-stage model shown includes a feature processing module and a quality classification module. The quality classification module includes an aggregation module, a first fully connected layer, a ReLU activation function, a second fully connected layer, and a softmax layer. The second data preprocessing module and the feature processing module perform data preprocessing, sequence feature acquisition, and feature fusion on the second training samples according to the process disclosed in the above embodiment, outputting second sample fusion features. These second sample fusion features are then input into the quality classification module for quality category prediction, outputting the predicted classification result. Subsequently, the second-stage model is trained based on the true quality category obtained by labeling the second training samples and the predicted result output by the quality classification module. Specifically, the aggregation module aggregates the sample fusion features output by the feature fusion module by taking the average, obtaining aggregated sample features. Then, the aggregated sample features are dimensionality-reduced through the first fully connected layer. The output of the first fully connected layer is sequentially passed through the ReLU activation function, the second fully connected layer, and the softmax layer to obtain the predicted classification result. The predicted classification result is the confidence score of each category. The confidence score of a category can be understood as the possible probability of that category, and the sum of the confidence scores of all categories is 1.

[0068] The training method for the video classification model provided in this disclosure performs multimodal self-supervised learning on a large number of unlabeled first training samples, which greatly improves the multimodal feature extraction capability of the feature processing module, enabling comprehensive and accurate sample feature extraction. Based on the feature processing module trained in the first training stage, supervised training for the downstream video quality classification task is performed on labeled second training samples, effectively improving the accuracy of video quality classification, thereby making the determination of video quality more accurate.

[0069] Based on the above embodiments, Figure 5 This is a flowchart illustrating a video quality determination method provided in an embodiment of the present disclosure, specifically including the following: Figure 5 Steps S510 to S530 are shown below:

[0070] S510. Obtain the target video and related text information of the target video.

[0071] The relevant text information is used to describe the video content of the target video. Specifically, the relevant text information of the target video may include the title information of the target video, a brief introduction of the video content, etc.

[0072] Understandably, before determining the quality of the target video, information in two different modalities—the target video itself and its related text information—is first obtained. Other modal information of the target video (such as audio information, editing template information, etc.) can also be determined by the user according to their needs and used as the basic data for determining the video quality. This is not limited here. Preferably, the target video can be a video from various fields such as automotive vertical short videos.

[0073] S520. The target video and the related text information are input into the data preprocessing module and then into the trained video classification model. After classification processing by the video classification model, the confidence level of each video quality category in the video classification model is output.

[0074] The confidence level is used to represent the probability that the target video belongs to the corresponding video quality category.

[0075] The video quality category may include at least three categories, including a first category, a second category, and a third category. Each category has a preset base score value. The at least three categories are sorted according to the size of their respective preset base scores values. The video quality level corresponding to each category is positively correlated with its corresponding preset base score value. That is, the higher the preset base score value of a category, the higher its corresponding video quality level, and the higher the video quality level of a category, the higher its preset score value.

[0076] Understandably, based on the above S510, the target video and its related text information are input into the data preprocessing module and then into the trained video classification model to predict the video quality category and obtain the confidence level corresponding to each video quality category.

[0077] In one optional implementation, the video quality categories set for the video classification model during model construction may include a first category, a second category, and a third category. The classification model may also have more than three quality categories. The first category can be considered a high-quality category, the second category a medium-quality category, and the third category a low-quality category. Specifically, the preset baseline score for the first category is greater than the preset baseline score for the second category, and the preset baseline score for the second category is greater than the preset baseline score for the third category. It is understood that during training, the video classification model labels the true quality category of the video samples as one of the three quality categories mentioned above. For example, the true quality category of video sample 1 is the medium-quality category. In application, after the video classification model predicts the quality category, it can output only the confidence score corresponding to each video quality category, without outputting the final predicted target quality category. Alternatively, it can output both the confidence score for each video quality category corresponding to the target video and the target quality category simultaneously.

[0078] Optionally, in step S520 above, the confidence level of each video quality category corresponding to the target video is obtained using a video classification model. This can be achieved through the following steps:

[0079] In the data preprocessing stage, frames are extracted from the target video according to a preset frame extraction frequency to obtain the video frame sequence corresponding to the target video. The relevant text information is serialized and encoded to obtain the text tag sequence corresponding to the target video. In the model inference stage, features are extracted from the video frame sequence to obtain the video modal sequence features of the target video. The text tag sequence is feature-mapped to obtain the text modal sequence features of the target video. The video modal sequence features and the text modal sequence features are fused to obtain fused features. Based on the fused features, video quality category prediction is performed to obtain the confidence level of each video quality category corresponding to the target video.

[0080] Understandably, after extracting frames from the target video at a preset frame extraction frequency, multiple video frames are obtained. These multiple video frames form a video frame sequence in playback order. The preset frame extraction frequency can be the same as the preset frequency involved in the aforementioned data preprocessing module. Subsequently, feature extraction is performed on the video frame sequence to obtain video modal sequence features. Relevant text information is serialized and encoded to obtain a text tag sequence. Then, feature mapping is performed on the text tag sequence to obtain text modal sequence features. The video modal sequence features and text modal sequence features are fused to obtain fused features that combine video and text modal information. Understandably, the above sequence feature acquisition and feature fusion are implemented based on the feature processing module in a pre-trained classification model. The specific implementation steps are described in the steps of the feature processing module of the aforementioned classification model, and will not be repeated here. After obtaining the fused features, the entire fused features are input into the quality classification module for quality category prediction, and the confidence score of each video quality category corresponding to the target video is output.

[0081] Optionally, the feature fusion process described above can be performed to obtain fused features, which can be achieved through the following steps:

[0082] The video modal sequence features and the text modal sequence features are concatenated to obtain concatenated features; the concatenated features are then subjected to multimodal feature interaction fusion processing to obtain the fused features of the target video.

[0083] Understandably, the video modal sequence features and text modal sequence features are first concatenated according to the set concatenation rules to obtain concatenated features. After obtaining the concatenated features, the Encoder module is used to perform multimodal feature interaction fusion on the concatenated features and output the fused features.

[0084] Optionally, the quality category prediction based on the fused features described above, to obtain the confidence level for each category, can be achieved through the following steps:

[0085] During the model inference stage, the fused features are aggregated by taking the mean to obtain aggregated features; the confidence level of each category corresponding to the target video is predicted based on the aggregated features.

[0086] Understandably, features are aggregated and fused by taking the average value to obtain aggregated features, and then the confidence level of each category is obtained through the quality classification module.

[0087] S530. Based on the confidence level of each video quality category corresponding to the target video, obtain the quality determination result of the target video;

[0088] The quality determination result is used to reflect the quality of the target video.

[0089] In one optional implementation, the confidence level of each video quality category corresponding to the target video and the preset base score value of each video quality category are fused and calculated to obtain the quality score value of the target video.

[0090] Understandably, existing video classification models, after determining the confidence level of a category, will directly use the category with the highest confidence level as the quality category of the target video and output it. In this case, it is impossible to further distinguish the more accurate quality differences between videos belonging to the same quality category.

[0091] Understandably, in response to the above technical problems, based on the above S520, after obtaining the confidence level of each video quality category corresponding to the target video, the confidence level of each video quality category and the preset base score value corresponding to each video quality category are weighted and summed to obtain the final quality score value of the target video. Understandably, different videos belonging to the same video quality category may have different qualities. The quality score value can further represent the quality difference between different videos in the same video quality category. For example, videos 1 and 2, which belong to the first category, have a quality score value of 0.8 and a quality score value of 0.75. The quality scores value can be used to sort the videos. It can be seen that the quality of video 1 is significantly better than that of video 2. That is, based on the more accurate quality score value, the quality of videos in the same category can be further distinguished, not limited to the quality category, and it is also convenient to conduct more accurate video quality review based on the quality score value in the future. The formula for calculating the quality score is as shown in the following formula (1):

[0092] score = c h *score h +c m *score m +c l *score l Formula (1)

[0093] In the formula, c h c m c l The scores represent the confidence levels for the first, second, and third categories, respectively. h score m score l These represent the preset base scores for the first, second, and third categories, respectively, where c h c m c l The sum of 1 is the preferred preset base score for each category. h =1.0, score m =0.5, scorel =0.1.

[0094] The video quality determination method provided in this disclosure uses information from both video and text modalities to predict the quality of a target video using a video classification model. Since both video and text modalities can comprehensively characterize the target video, this disclosure can more accurately determine its quality. Furthermore, this disclosure improves the granularity of the quality category labels, further enhancing the accuracy of video quality determination.

[0095] Furthermore, this embodiment adds at least one additional quality category to the binary classification, and models the quality scoring problem as a method of calculating scores based on the confidence scores of each quality category. After acquiring information about the target video in both video and text modalities and performing data preprocessing and model inference, the confidence score of each video quality category corresponding to the target video output by the video classification model is obtained. Based on the confidence scores of each video quality category corresponding to the target video, a fusion calculation is performed to obtain the final quality score. By calculating the quality score of the target video, the accuracy of video quality determination can be improved, and it is not limited by the inherent quality categories of the model. For different videos within the same quality category, further differentiation can be achieved through quality score values.

[0096] Figure 6 This is a schematic diagram of a video quality determination device provided in an embodiment of the present disclosure. The quality determination device provided in this embodiment can execute the processing flow provided in the video quality determination method embodiment, such as… Figure 6 As shown, the video quality determination device 600 includes a first acquisition unit 610, a model processing unit 620, and a second acquisition unit 630, wherein:

[0097] The first acquisition unit 610 is used to acquire a target video and related text information of the target video; wherein, the related text information is used to describe the video content of the target video;

[0098] The model processing unit 620 is used to input the preprocessed target video and the related text information into a trained video classification model, and after classification processing by the video classification model, output the confidence score of each video quality category in the video classification model; wherein, the confidence score is used to represent the probability that the target video belongs to the corresponding video quality category;

[0099] The second acquisition unit 630 is used to acquire the quality determination result of the target video based on the confidence level of each video quality category corresponding to the target video.

[0100] Optionally, the second acquisition unit 630 includes:

[0101] The fusion calculation subunit is used to fuse the confidence level of each video quality category corresponding to the target video with the preset base score value corresponding to each video quality category to obtain the quality score value of the target video.

[0102] The first determining subunit is used to determine the quality score value as the quality determination result of the target video.

[0103] Optional, preprocessing module, used for:

[0104] Frames are extracted from the target video to obtain a video frame sequence corresponding to the target video, and the relevant text information is serialized and encoded to obtain a text tag sequence corresponding to the target video;

[0105] Accordingly, the model processing unit 620 includes:

[0106] The feature acquisition subunit is used to input the video frame sequence and text tag sequence corresponding to the target video into a trained video classification model, use the video classification model to extract features from the video frame sequence to obtain the video modal sequence features of the target video, and perform feature mapping on the text tag sequence to obtain the text modal sequence features of the target video.

[0107] The feature fusion subunit is used to perform feature fusion processing on the video modal sequence features and the text modal sequence features to obtain fused features;

[0108] The category prediction subunit is used to predict the quality category based on the fused features and obtain the confidence level of each category.

[0109] Optional feature fusion subunits include:

[0110] The first splicing subunit is used to splice the video modal sequence features and the text modal sequence features to obtain spliced ​​features;

[0111] The first fusion processing submodule is used to perform multimodal feature interaction fusion processing on the spliced ​​features to obtain the fused features of the target video.

[0112] Optional, category prediction subunit, used for:

[0113] The fused features are aggregated by taking the average value to obtain aggregated features.

[0114] The confidence level of each video quality category corresponding to the target video is obtained based on the aggregated features.

[0115] Optionally, device 600 also includes:

[0116] The third acquisition unit is used to acquire a first training sample set and a second training sample set; wherein, the first training sample set includes a first video training sample and a first text training sample with a corresponding relationship, the first text training sample is used to describe the video content of the corresponding first video training sample, and the second training sample set includes a second video training sample, a second text training sample and a video quality category label with a corresponding relationship, the second text training sample is used to describe the video content of the corresponding second video training sample.

[0117] The first training unit is used to perform self-supervised training on the first-stage model using the first training sample set to obtain the trained first-stage model. The first-stage model includes a feature processing module and learning modules corresponding to each preset learning task. The first data preprocessing module and the feature processing module corresponding to the first-stage model are used to generate real sample labels for each preset learning task. The feature processing module is used to generate first sample fusion features based on the preprocessing results of the first data preprocessing module. The learning module is used to output prediction results for the corresponding preset learning task. The prediction results of each preset learning task and the real sample labels of each preset learning task are used to update the model parameters of the first-stage model.

[0118] The second training unit is used to construct a second-stage model based on the quality classification module and the feature processing module in the trained first-stage model, and to perform supervised training on the second-stage model using the second training sample set to obtain the trained second-stage model, which serves as the trained video classification model.

[0119] Optionally, the first training unit includes:

[0120] The first sample preprocessing subunit is used to input the first training sample set into the first data preprocessing module corresponding to the first stage model to obtain the preprocessing result; the preprocessing result includes the real sample labels of the text modality, the text modality mask label sequence, and the video frame sequence:

[0121] The first training processing subunit is used to input the text modality mask label sequence and the video frame sequence into the feature processing module in the first stage model. After processing by the feature processing module, the first sample fusion feature and the real sample label of the video modality are obtained.

[0122] The first prediction subunit is used to obtain the sample prediction results of each preset learning task based on the first sample fusion features and by using the learning modules corresponding to each preset learning task.

[0123] The parameter update subunit is used to update the model parameters of the first-stage model based on the sample prediction results of each preset learning task and the real sample labels of each preset learning task; the real sample labels of each preset learning task include the real sample labels of the text modality and the real sample labels of the video modality.

[0124] Optionally, the first training processing subunit includes:

[0125] The first encoding subunit is used to use the first data preprocessing module corresponding to the first stage model to perform serialization encoding on the first text training sample in the first training sample set to obtain the text real sample label sequence of the first text training sample, which serves as the real sample label of the text modality.

[0126] The first masking subunit is used to mask the text markers at the positions of the sequences to be predicted in the real sample labels of the text modality to obtain a text modality mask marker sequence, and to obtain text mask sample features by performing feature mapping on the text modality mask marker sequence.

[0127] Optionally, the first training processing subunit includes:

[0128] The second masking subunit is used to extract features from the video frame sequence to obtain real video sample features, and to perform masking processing on the features of the sequence position to be predicted in the real video sample features to generate video masked sample features.

[0129] The feature mapping subunit is used to perform feature mapping on the text modality mask marker sequence to obtain text mask sample features;

[0130] The second splicing subunit is used to splice the video mask sample features and the text mask sample features according to the set splicing rules to obtain spliced ​​mask sample features.

[0131] The second fusion processing subunit is used to perform multimodal feature interaction fusion processing on the spliced ​​mask sample features to obtain the first sample fusion features.

[0132] Optionally, the first prediction subunit includes:

[0133] The input feature determination subunit is used to, when the preset learning task includes a video learning task, utilize the learning module corresponding to the preset learning task to obtain the sequence position of the video mask sample feature in the spliced ​​mask sample feature, and determine the input feature of the learning module corresponding to the video learning task in the first sample fusion feature based on the sequence position; when the preset learning task includes a text learning task, determine the first sample fusion feature as the input feature of the learning module corresponding to the text learning task.

[0134] The learning processing subunit is used to input the input features into the corresponding learning module, and after processing by the learning module, obtain the sample prediction result of the corresponding preset learning task.

[0135] Optional, the second training unit includes:

[0136] The fusion processing subunit is used to perform data preprocessing, sequence feature acquisition and feature fusion on the second video training samples and the second text training samples with corresponding relationships in the second training sample set by using the second data preprocessing module and the feature processing module corresponding to the second stage model, so as to obtain the second sample fusion feature.

[0137] The quality classification processing subunit is used to input the fused features of the second sample into the quality classification module in the second stage model, and after processing by the quality classification module, output the classification prediction result of the second video training sample;

[0138] The loss calculation subunit is used to calculate the loss value based on the classification prediction result and the video quality category label of the second video training sample.

[0139] The model parameter update subunit is used to update the model parameters of the second-stage model according to the loss value until the model convergence condition is met, thus obtaining the trained video classification model.

[0140] Optionally, the video quality categories include at least three categories sorted according to the size of a preset base score, and the video quality level corresponding to each of the at least three categories is positively correlated with the preset base score.

[0141] Figure 6 The video quality determination device shown in the embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0142] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. See below for details. Figure 7The diagram illustrates a structural schematic suitable for implementing the electronic device 700 in the embodiments of this disclosure. The electronic device 700 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, smart home devices, etc. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0143] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703 to implement the video quality determination method as described in the embodiments of this disclosure. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing device 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0144] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0145] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the video quality determination method as described above. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this disclosure.

[0146] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0147] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0148] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0149] Optionally, when one or more of the above-described procedures are executed by the electronic device, the electronic device may also perform other steps described in the above embodiments.

[0150] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0152] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0153] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0154] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0155] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or gateway that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or gateway. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or gateway that includes said element.

[0156] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for determining video quality, characterized in that, include: Obtain the target video and related text information of the target video; wherein, the related text information is used to describe the video content of the target video; The preprocessed target video and related text information are input into a trained video classification model. After classification by the video classification model, the confidence score of each video quality category in the video classification model is output; wherein, the confidence score is used to represent the probability that the target video belongs to the corresponding video quality category. Based on the confidence level of each video quality category corresponding to the target video, the quality determination result of the target video is obtained; The video classification model is trained through the following steps: Obtain a first training sample set and a second training sample set; wherein, the first training sample set includes a first video training sample and a first text training sample with a corresponding relationship, and the second training sample set includes a second video training sample, a second text training sample and a video quality category label with a corresponding relationship, and the second text training sample is used to describe the video content of the corresponding second video training sample; The first-stage model is self-supervised and trained using the first training sample set to obtain the trained first-stage model. A second-stage model is constructed based on the quality classification module and the feature processing module in the trained first-stage model. The second-stage model is then trained in a supervised manner using the second training sample set to obtain the trained second-stage model, which serves as the trained video classification model.

2. The method according to claim 1, characterized in that, The process of obtaining the quality determination result of the target video based on the confidence level of each video quality category corresponding to the target video includes: The confidence level of each video quality category corresponding to the target video is fused with the preset base score value corresponding to each video quality category to obtain the quality score value of the target video. The quality score is determined as the quality determination result of the target video.

3. The method according to claim 1, characterized in that, Before inputting the preprocessed target video and the related text information into the trained video classification model, and before outputting the confidence score of each video quality category in the video classification model after classification processing by the video classification model, the method further includes: Frames are extracted from the target video to obtain a video frame sequence corresponding to the target video, and the relevant text information is serialized and encoded to obtain a text tag sequence corresponding to the target video; Accordingly, the preprocessed target video and related text information are input into a trained video classification model. After classification by the video classification model, the confidence score of each video quality category in the video classification model is output, including: The video frame sequence and text tag sequence corresponding to the target video are input into a trained video classification model. The video classification model is used to extract features from the video frame sequence to obtain the video modal sequence features of the target video, and to perform feature mapping on the text tag sequence to obtain the text modal sequence features of the target video. Using the video classification model, the video modal sequence features and the text modal sequence features are fused to obtain the fused features of the target video. Based on the fused features, the video quality category is predicted to obtain the confidence level of each video quality category corresponding to the target video.

4. The method according to claim 3, characterized in that, The step of fusing the video modal sequence features and the text modal sequence features to obtain the fused features of the target video includes: The video modal sequence features and the text modal sequence features are concatenated to obtain concatenated features; The spliced ​​features are subjected to multimodal feature interaction fusion processing to obtain the fused features of the target video.

5. The method according to claim 3 or 4, characterized in that, The step of predicting video quality category based on the fused features to obtain the confidence level of each video quality category corresponding to the target video includes: The fused features are aggregated by taking the average value to obtain aggregated features. The confidence level of each video quality category corresponding to the target video is obtained based on the aggregated features.

6. The method according to claim 1, characterized in that, The first text training sample is used to describe the video content corresponding to the first video training sample; the first stage model includes a feature processing module and a learning module corresponding to each preset learning task. The first data preprocessing module and the feature processing module corresponding to the first stage model are used to generate real sample labels for each preset learning task. The feature processing module is used to generate first sample fusion features based on the preprocessing results of the first data preprocessing module. The learning module is used to output the prediction results corresponding to the preset learning task. The prediction results of each preset learning task and the real sample labels of each preset learning task are used to update the model parameters of the first stage model.

7. The method according to claim 6, characterized in that, The step of performing self-supervised training on the first-stage model using the first training sample set to obtain the trained first-stage model includes: The first training sample set is input into the first data preprocessing module corresponding to the first stage model to obtain the preprocessing result; the preprocessing result includes the real sample labels of the text modality, the text modality mask marker sequence, and the video frame sequence; The text modality masking sequence and the video frame sequence are input into the feature processing module in the first stage model. After processing by the feature processing module, the first sample fusion feature and the real sample label of the video modality are obtained. Based on the first sample fusion feature, the sample prediction results of each preset learning task are obtained by using the learning modules corresponding to each preset learning task. Based on the sample prediction results of each preset learning task and the real sample labels of each preset learning task, the model parameters of the first stage model are updated; the real sample labels of each preset learning task include the real sample labels of the text modality and the real sample labels of the video modality.

8. The method according to claim 7, characterized in that, The step of inputting the first training sample set into the first data preprocessing module corresponding to the first stage model to obtain the preprocessing result includes: Using the first data preprocessing module corresponding to the first stage model, the first text training sample in the first training sample set is serialized and encoded to obtain the text real sample label sequence of the first text training sample, which is used as the real sample label of the text modality. Furthermore, the text markers at the positions of the sequences to be predicted in the real sample labels of the text modality are masked to obtain a text modality mask marker sequence, and the text mask sample features are obtained by feature mapping on the text modality mask marker sequence.

9. The method according to claim 8, characterized in that, The text modality masking sequence and the video frame sequence are input into the feature processing module of the first-stage model. After processing by the feature processing module, the first sample fusion feature and the true sample label of the video modality are obtained, including: Feature extraction is performed on the video frame sequence to obtain real video sample features. The features of the sequence position to be predicted in the real video sample features are then masked to generate video masked sample features. Feature mapping is performed on the text modality mask marker sequence to obtain text mask sample features; The video mask sample features and the text mask sample features are concatenated according to the set concatenation rules to obtain the concatenated mask sample features. The spliced ​​mask sample features are subjected to multimodal feature interaction fusion processing to obtain the first sample fusion feature.

10. The method according to claim 9, characterized in that, The step of obtaining sample prediction results for each preset learning task based on the first sample fusion feature and utilizing the learning modules corresponding to each preset learning task includes: When the preset learning task includes a video learning task, the learning module corresponding to the preset learning task is used to obtain the sequence position of the video mask sample feature in the spliced ​​mask sample feature, and the input feature of the learning module corresponding to the video learning task is determined in the first sample fusion feature based on the sequence position. When the preset learning task includes a text learning task, the first sample fusion feature is determined as the input feature of the learning module corresponding to the text learning task; The input features are fed into the corresponding learning module, and after processing by the learning module, the sample prediction results for the corresponding preset learning task are obtained.

11. The method according to claim 6, characterized in that, The step of using the second training sample set to perform supervised training on the second-stage model to obtain the trained second-stage model, which serves as the trained video classification model, includes: Using the second data preprocessing module and the feature processing module corresponding to the second stage model, data preprocessing, sequence feature acquisition and feature fusion are performed on the corresponding second video training samples and second text training samples in the second training sample set to obtain the second sample fusion feature. The second sample fusion feature is input into the quality classification module in the second stage model. After processing by the quality classification module, the classification prediction result of the second video training sample is output. The loss value is calculated based on the classification prediction result and the video quality category label of the second video training sample; The model parameters of the second-stage model are updated based on the loss value until the model convergence condition is met, resulting in a trained video classification model.

12. The method according to claim 2, characterized in that, The video quality categories include at least three categories sorted according to the size of a preset base score, and the video quality level corresponding to each of the at least three categories is positively correlated with the preset base score.

13. A video quality determination device, characterized in that, include: The first acquisition unit is used to acquire a target video and related text information of the target video; wherein, the related text information is used to describe the video content of the target video; The model processing unit is used to input the preprocessed target video and the related text information into a trained video classification model, and after classification processing by the video classification model, output the confidence score of each video quality category in the video classification model; wherein, the confidence score is used to represent the probability that the target video belongs to the corresponding video quality category; The second acquisition unit is used to acquire the quality determination result of the target video based on the confidence level of each video quality category corresponding to the target video; The video classification model is trained through the following steps: Obtain a first training sample set and a second training sample set; wherein, the first training sample set includes a first video training sample and a first text training sample with a corresponding relationship, and the second training sample set includes a second video training sample, a second text training sample and a video quality category label with a corresponding relationship, and the second text training sample is used to describe the video content of the corresponding second video training sample; The first-stage model is self-supervised and trained using the first training sample set to obtain the trained first-stage model. A second-stage model is constructed based on the quality classification module and the feature processing module in the trained first-stage model. The second-stage model is then trained in a supervised manner using the second training sample set to obtain the trained second-stage model, which serves as the trained video classification model.

14. An electronic device, characterized in that, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the video quality determination method as described in any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the video quality determination method as described in any one of claims 1 to 12.

Images