Video quality evaluation model training, video quality determination method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a video quality evaluation model training method, a video quality determination method, a device, an electronic device and a storage medium. The method comprises: performing feature extraction on a plurality of video samples based on a first model to obtain first feature information of each video sample, performing feature extraction on the plurality of video samples based on a second model to obtain second feature information of each video sample, performing feature dimension transformation and feature information extraction on the first feature information of each video sample and the second feature information of each video sample based on a feature dimension transformation module in a quality evaluation model to determine comprehensive feature information of each video sample, training the quality evaluation model using the first feature information, the second feature information, the comprehensive feature information and quality label information of each video sample to obtain a trained quality evaluation model. The present application can use various models to help improve the accuracy of the video quality evaluation task through the quality-related information contained therein.

Description

Technical Field

[0001] This disclosure relates to the field of Internet technology, and in particular to a video quality assessment model training, video quality determination method, apparatus, electronic device and storage medium. Background Technology

[0002] In recent years, video-focused social networking platforms have become incredibly popular. This surge in video content consumption has presented video providers with a significant challenge in offering better services. Since the perceived quality of video significantly impacts the overall experience, identifying video quality has become one of the most crucial issues.

[0003] Video Quality Assessment (VQA) mimics the subjective feedback humans give when watching videos, aiming to automatically evaluate the perceived quality of input videos. It has been extensively studied in assessing compression artifacts, transmission errors, and overall quality. Compared to traditional methods based on handcrafted features, data-driven deep learning (DL-based) methods have shown better performance and are attracting increasing attention. However, the relatively small datasets for VQA compared to other high-level visual tasks limit the application of deep learning methods. This is because VQA is a highly subjective task. Obtaining unbiased annotation information (MOS) requires multiple annotations and cross-validation. For example, the KoNViD-1k dataset requires an average of 114 annotations per video. The high cost of annotation limits the size of VQA datasets. To address this issue, existing methods attempt to fine-tune pre-trained models; however, given the numerous factors influencing VQA, current methods cannot accurately assess video quality. Summary of the Invention

[0004] This disclosure provides a video quality assessment model training method, device, electronic device, and storage medium, as well as a video quality determination method, apparatus, electronic device, and storage medium. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, a method for training a video quality assessment model is provided, comprising:

[0006] Feature extraction is performed on multiple video samples based on the first model to obtain the first feature information of each video sample; feature extraction is performed on multiple video samples based on the second model to obtain the second feature information of each video sample; each video sample carries quality annotation information; the first feature information is the information corresponding to the video segment; the second feature information is the information corresponding to the video frame;

[0007] Based on the feature dimension transformation module in the quality assessment model, feature dimension transformation and feature information extraction are performed on the first feature information and the second feature information of each video sample to determine the comprehensive feature information of each video sample.

[0008] The quality assessment model is trained using the intra-sample consistency loss function, the inter-sample difference loss function, and the sample true label difference loss function, as well as the first feature information, second feature information, comprehensive feature information, and quality annotation information of each video sample, to obtain the trained quality assessment model.

[0009] The intra-sample consistency loss function is used to constrain the feature information of the same video sample under different types of models to be in a consistent feature latent space; the inter-sample difference loss function is used to constrain the feature information of video samples carrying different quality annotation information to be in different feature latent spaces under the same model.

[0010] In some possible embodiments, based on the feature dimension transformation module in the quality assessment model, feature dimension transformation and feature information extraction are performed on the first feature information and the second feature information of each video sample to determine the comprehensive feature information of each video sample, including:

[0011] The feature dimension transformation module is used to transform the first feature information and extract the feature information of each video sample, respectively, to obtain the first feature information and the second feature information of each video sample with the same feature dimension.

[0012] The first and second feature information with the same feature dimension for each video sample are aggregated to obtain the comprehensive feature information for each video sample.

[0013] In some possible embodiments, the quality assessment model also includes a weight allocation module;

[0014] The quality assessment model is trained using the intra-sample consistency loss function, the inter-sample dissimilarity loss function, and the sample true label dissimilarity loss function, along with the first feature information, second feature information, comprehensive feature information, and quality annotation information of each video sample. The trained quality assessment model includes:

[0015] Based on the weight allocation module, the weight data corresponding to each sub-feature information in the comprehensive feature information is used to weight each sub-feature information to obtain the target feature information;

[0016] The quality prediction information of each video sample is obtained by using the sample true label difference loss function to predict the target feature information of each video sample.

[0017] The quality loss data for each video sample is determined based on the quality prediction information and quality identification information of each video sample.

[0018] The first and second feature information of each video sample are processed using the intra-sample consistency loss function to determine the intra-sample loss data for each video sample.

[0019] The first and second feature information of each video sample are processed using the inter-sample difference loss function to determine the inter-sample loss data.

[0020] The quality assessment model is trained using quality loss data, intra-sample loss data, and inter-sample loss data to obtain the trained quality assessment model.

[0021] In some possible embodiments, the first and second feature information of each video sample are processed using an inter-sample discrepancy loss function to determine the inter-sample loss data, including:

[0022] The first inter-sample loss data is determined by processing two first feature information of any two video samples carrying different quality annotation information in the output of the first model using the inter-sample difference loss function; or, the first inter-sample loss data is determined by processing two second feature information of any two video samples carrying different quality annotation information in the output of the second model using the inter-sample difference loss function.

[0023] The inter-sample loss function is used to process two first feature information of any two video samples carrying the same quality annotation information output by the first model to determine the second inter-sample loss data; or, the inter-sample loss function is used to process two second feature information of any two video samples carrying the same quality annotation information output by the second model to determine the second inter-sample loss data.

[0024] The inter-sample loss data is determined based on the first inter-sample loss data and the second inter-sample loss data.

[0025] In some possible embodiments, the quality assessment model is trained using quality loss data, intra-sample loss data, and between-sample loss data to obtain a trained quality assessment model, including:

[0026] The quality assessment model is trained using quality loss data, intra-sample loss data, and inter-sample loss data until, for the same video sample, the first similarity data between the first feature information output by the first model and the second feature information output by the second model in the quality assessment model is less than the first preset similarity data; until, for two video samples carrying different quality annotation information, the second similarity data between the two first feature information output by the first model in the quality assessment model is greater than the second preset similarity data; until, for two video samples carrying the same quality annotation information, the third similarity data between the two second feature information output by the second model in the trained quality assessment model is less than the third preset similarity data; the third preset similarity data is less than the second preset similarity data; and until, the fourth similarity data between the predicted annotation information determined based on the comprehensive feature information of each video sample and the quality annotation information of each video sample is less than the fourth preset similarity data, then the model training stops, and the trained quality assessment model is obtained.

[0027] In some possible embodiments, the method further includes:

[0028] Multiple models are identified; each model is used to process at least one original task; the feature representations obtained by each model in processing the original task are related to video quality.

[0029] Among multiple models, the model related to video segment processing is identified as the first model;

[0030] Among the multiple models, the one related to video frame processing is identified as the second model.

[0031] In some possible embodiments, feature extraction is performed on multiple video samples based on a first model to obtain first feature information for each video sample, including:

[0032] Frame extraction is performed on each video sample from multiple video samples to obtain a preset number of image frames for each video sample.

[0033] The image frames of each video sample are sorted according to their sequential position within the video sample to obtain a video segment for each video sample.

[0034] Based on the first model, feature extraction is performed on the video segments of each video sample to obtain the first feature information of each video sample.

[0035] In some possible embodiments, feature extraction is performed on multiple video samples based on a second model to obtain second feature information for each video sample, including:

[0036] Frame extraction is performed on each video sample from multiple video samples to obtain a preset number of image frames for each video sample.

[0037] Based on the second model, feature extraction is performed on each image frame in a preset number of image frames for each video sample to obtain sub-feature information of each image frame in a preset number of image frames for each video sample.

[0038] The second feature information of each video sample is determined based on the sub-feature information of each image frame in a preset number of image frames for each video sample.

[0039] According to a second aspect of the present disclosure, a video quality determination method is provided, comprising:

[0040] Confirm the video to be confirmed;

[0041] Based on the first model, feature extraction is performed on the video to be determined to obtain the first feature information of the video to be determined.

[0042] Based on the second model, feature extraction is performed on the video to be determined to obtain the second feature information of the video to be determined.

[0043] The first and second feature information of the video to be determined are input into the trained quality assessment model to obtain the quality data of the video to be determined.

[0044] According to a third aspect of the present disclosure, a video quality assessment model training apparatus is provided, comprising:

[0045] The feature extraction module is configured to perform feature extraction on multiple video samples based on a first model to obtain first feature information for each video sample; and to perform feature extraction on multiple video samples based on a second model to obtain second feature information for each video sample; each video sample carries quality annotation information; the first feature information is the information corresponding to the video segment; and the second feature information is the information corresponding to the video frame.

[0046] The feature determination module is configured to perform feature dimension transformation and feature information extraction on the first feature information and the second feature information of each video sample based on the feature dimension transformation module in the quality assessment model, and determine the comprehensive feature information of each video sample.

[0047] The model training module is configured to train the quality assessment model using the intra-sample consistency loss function, the inter-sample difference loss function, and the sample true label difference loss function, as well as the first feature information, second feature information, comprehensive feature information, and quality annotation information of each video sample, to obtain the trained quality assessment model.

[0048] The intra-sample consistency loss function is used to constrain the feature information of the same video sample under different types of models to be in a consistent feature latent space; the inter-sample difference loss function is used to constrain the feature information of video samples carrying different quality annotation information to be in different feature latent spaces under the same model.

[0049] In some possible embodiments, the feature determination module is configured to perform:

[0050] The feature dimension transformation module is used to transform the first feature information and extract the feature information of each video sample, respectively, to obtain the first feature information and the second feature information of each video sample with the same feature dimension.

[0051] The first and second feature information with the same feature dimension for each video sample are aggregated to obtain the comprehensive feature information for each video sample.

[0052] In some possible embodiments, the quality assessment model also includes a weight allocation module;

[0053] The model training module is configured to execute:

[0054] Based on the weight allocation module, the weight data corresponding to each sub-feature information in the comprehensive feature information is used to weight each sub-feature information to obtain the target feature information;

[0055] The quality prediction information of each video sample is obtained by using the sample true label difference loss function to predict the target feature information of each video sample.

[0056] The quality loss data for each video sample is determined based on the quality prediction information and quality identification information of each video sample.

[0057] The first and second feature information of each video sample are processed using the intra-sample consistency loss function to determine the intra-sample loss data for each video sample.

[0058] The first and second feature information of each video sample are processed using the inter-sample difference loss function to determine the inter-sample loss data.

[0059] The quality assessment model is trained using quality loss data, intra-sample loss data, and inter-sample loss data to obtain the trained quality assessment model.

[0060] In some possible embodiments, the model training module is configured to perform:

[0061] The first inter-sample loss data is determined by processing two first feature information of any two video samples carrying different quality annotation information in the output of the first model using the inter-sample difference loss function; or, the first inter-sample loss data is determined by processing two second feature information of any two video samples carrying different quality annotation information in the output of the second model using the inter-sample difference loss function.

[0062] The inter-sample loss function is used to process two first feature information of any two video samples carrying the same quality annotation information output by the first model to determine the second inter-sample loss data; or, the inter-sample loss function is used to process two second feature information of any two video samples carrying the same quality annotation information output by the second model to determine the second inter-sample loss data.

[0063] The inter-sample loss data is determined based on the first inter-sample loss data and the second inter-sample loss data.

[0064] In some possible embodiments, the model training module is configured to perform:

[0065] The quality assessment model is trained using quality loss data, intra-sample loss data, and inter-sample loss data until, for the same video sample, the first similarity data between the first feature information output by the first model and the second feature information output by the second model in the quality assessment model is less than the first preset similarity data; until, for two video samples carrying different quality annotation information, the second similarity data between the two first feature information output by the first model in the quality assessment model is greater than the second preset similarity data; until, for two video samples carrying the same quality annotation information, the third similarity data between the two second feature information output by the second model in the trained quality assessment model is less than the third preset similarity data; the third preset similarity data is less than the second preset similarity data; and until, the fourth similarity data between the predicted annotation information determined based on the comprehensive feature information of each video sample and the quality annotation information of each video sample is less than the fourth preset similarity data, then the model training stops, and the trained quality assessment model is obtained.

[0066] In some possible embodiments, the apparatus further includes a model determination module configured to perform:

[0067] Multiple models are identified; each model is used to process at least one original task; the feature representations obtained by each model in processing the original task are related to video quality.

[0068] Among multiple models, the model related to video segment processing is identified as the first model;

[0069] Among the multiple models, the one related to video frame processing is identified as the second model.

[0070] In some possible embodiments, the feature extraction module is configured to perform:

[0071] Frame extraction is performed on each video sample from multiple video samples to obtain a preset number of image frames for each video sample.

[0072] The image frames of each video sample are sorted according to their sequential position within the video sample to obtain a video segment for each video sample.

[0073] Based on the first model, feature extraction is performed on the video segments of each video sample to obtain the first feature information of each video sample.

[0074] In some possible embodiments, the feature extraction module is configured to perform:

[0075] Frame extraction is performed on each video sample from multiple video samples to obtain a preset number of image frames for each video sample.

[0076] Based on the second model, feature extraction is performed on each image frame in a preset number of image frames for each video sample to obtain sub-feature information of each image frame in a preset number of image frames for each video sample.

[0077] The second feature information of each video sample is determined based on the sub-feature information of each image frame in a preset number of image frames for each video sample.

[0078] According to a fourth aspect of the present disclosure, a video quality determination apparatus is provided, comprising:

[0079] The video determination module is configured to determine the video to be determined.

[0080] The first extraction module is configured to perform feature extraction on the video to be determined based on the first model, and obtain the first feature information of the video to be determined.

[0081] The second extraction module is configured to perform feature extraction on the video to be determined based on the second model, and obtain the second feature information of the video to be determined.

[0082] The quality determination module is configured to input the first and second feature information of the video to be determined into the trained quality assessment model to obtain the quality data of the video to be determined.

[0083] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement the method as described in any of the first aspects above or the method in the second aspect.

[0084] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided such that, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the method of any one of the first aspects of the present disclosure or the method of the second aspect.

[0085] According to a seventh aspect of the present disclosure, a computer program product is provided, the computer program product including a computer program stored in a readable storage medium, wherein at least one processor of a computer device reads from the readable storage medium and executes the computer program, causing the computer device to perform the method of any one of the first aspects of the present disclosure or the method of the second aspect.

[0086] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0087] Feature extraction is performed on multiple video samples based on a first model to obtain the first feature information for each video sample. Feature extraction is then performed on multiple video samples based on a second model to obtain the second feature information for each video sample. Each video sample carries quality labeling information. The first feature information corresponds to the video segment, and the second feature information corresponds to the video frame. The feature dimension transformation module in the quality assessment model performs feature dimension transformation and feature extraction on the first and second feature information of each video sample to determine the comprehensive feature information of each video sample. The quality assessment model is trained using the intra-sample consistency loss function, the inter-sample difference loss function, and the sample true label difference loss function, along with the first, second, comprehensive, and quality labeling information of each video sample, to obtain the trained quality assessment model. This application can use various pre-trained models, leveraging the quality-related information they contain, and new models to help improve the accuracy of video quality assessment tasks without changing the structure of the pre-trained models.

[0088] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0089] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0090] Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment;

[0091] Figure 2 This is a flowchart illustrating a video quality assessment model training method according to an exemplary embodiment;

[0092] Figure 3 This is a flowchart illustrating a method for determining first feature information according to an exemplary embodiment;

[0093] Figure 4 This is a flowchart illustrating a method for determining second feature information according to an exemplary embodiment;

[0094] Figure 5 This is a flowchart illustrating a method for determining comprehensive feature information according to an exemplary embodiment;

[0095] Figure 6 This is a flowchart illustrating a method for determining a trained quality assessment model according to an exemplary embodiment;

[0096] Figure 7 This is a flowchart illustrating a video quality determination method according to an exemplary embodiment;

[0097] Figure 8 This is a block diagram of a video quality assessment model training device according to an exemplary embodiment;

[0098] Figure 9 This is a block diagram illustrating a video quality determination apparatus according to an exemplary embodiment;

[0099] Figure 10 This is a block diagram illustrating an electronic device for training a video quality assessment model or determining video quality according to an exemplary embodiment. Detailed Implementation

[0100] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0101] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar first objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0102] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0103] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application environment for a video quality assessment model training method according to an exemplary embodiment, such as... Figure 1 As shown, the application environment may include server 01 and client 02.

[0104] In some possible embodiments, server 01 may be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud audio recognition model training, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The operating system running on the server may include, but is not limited to, Android, iOS, Linux, Windows, Unix, etc.

[0105] In some possible embodiments, the client 02 described above may include, but is not limited to, smartphones, desktop computers, tablets, laptops, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, smart wearable devices, and other similar clients. It may also be software running on the aforementioned client, such as applications or mini-programs. Optionally, the operating system running on the client may include, but is not limited to, Android, iOS, Linux, Windows, Unix, and other similar systems.

[0106] In some possible embodiments, client 02 or server 01 can extract features from multiple video samples based on a first model to obtain first feature information for each video sample, and then extract features from multiple video samples based on a second model to obtain second feature information for each video sample. Each video sample carries quality labeling information; the first feature information corresponds to a video segment, and the second feature information corresponds to a video frame. Based on the feature dimension transformation module in the quality assessment model, feature dimension transformation and feature information extraction are performed on the first and second feature information of each video sample to determine the comprehensive feature information of each video sample. The quality assessment model is then trained using the intra-sample consistency loss function, the inter-sample difference loss function, and the sample true label difference loss function, as well as the first, second, comprehensive, and quality labeling information of each video sample, to obtain the trained quality assessment model. This application can use various pre-trained models, leveraging the quality-related information they contain and new models, to help improve the accuracy of video quality assessment tasks without changing the structure of the pre-trained models.

[0107] In some possible embodiments, client 02 can be connected to server 01 via a wired link or a wireless link.

[0108] In one exemplary implementation, the client, server, and corresponding database can all be node devices in the blockchain system, capable of sharing acquired and generated information with other node devices in the blockchain system, thus achieving information sharing among multiple node devices. Multiple node devices in the blockchain system can be configured with the same blockchain, which consists of multiple blocks, and adjacent blocks are related, ensuring that any data tampering in any block can be detected by the next block, thereby preventing data tampering in the blockchain and guaranteeing the security and reliability of the data in the blockchain.

[0109] Figure 2 This is a flowchart illustrating a video quality assessment model training method according to an exemplary embodiment, such as... Figure 2 As shown, the video quality assessment model training method can be applied to servers or other node devices, such as clients. The following explanation uses a server as an example. This method includes at least the following steps S201 to S205:

[0110] In step S201, feature extraction is performed on multiple video samples based on the first model to obtain the first feature information of each video sample; feature extraction is performed on multiple video samples based on the second model to obtain the second feature information of each video sample; each video sample carries quality annotation information; the first feature information is the information corresponding to the video segment; the second feature information is the information corresponding to the video frame.

[0111] In this embodiment of the application, before performing feature extraction on multiple video samples based on the first model to obtain the first feature information of each video sample, and performing feature extraction on multiple video samples based on the second model to obtain the second feature information of each video sample, it is also necessary to determine the first model and the second model.

[0112] In one possible embodiment, the server may determine multiple models. Each model is used to process at least one original task, and the feature representations acquired by each model in processing the original task are related to the quality of the video.

[0113] In one alternative implementation, each model can handle one original task, such as model A handling original task A, model B handling original task B, model C handling original task C, and so on. That is, the original tasks handled by each model can be different and the number can be one.

[0114] In another alternative implementation, each model can handle multiple original tasks, and there may be two models that can handle the same original tasks. For example, model A can handle original task A and original task B, and model B can handle original task B and original task C.

[0115] In this embodiment, since the training model's role is to determine the quality of the video to be evaluated using the final quality assessment model, the feature representations acquired by each model during the original task can be related to video quality. Optionally, some models' original task is a video classification task, in which the model can acquire information related to motion, blur, or jitter related to video quality during video classification. Optionally, some models' original task is an image-text retrieval and matching task, in which the model can acquire image content information related to video quality during image-text retrieval. Optionally, some models' original tasks are image classification tasks, image target object recognition tasks, etc.

[0116] In other words, in this embodiment, multiple models can handle different original tasks, and each model has its own original tasks that are related to video quality. Under this premise, this embodiment can be retrained based on different models with specific feature acquisition, thereby obtaining the final quality assessment model while reducing the resources consumed in the entire training process.

[0117] In this embodiment, the multiple models may include a first model of a first type and a second model of a second type. The first and second models can process the video in different ways to obtain multi-dimensional video features, laying the foundation for subsequent video quality evaluation.

[0118] In one alternative implementation, the first model of the first type can be one or more, such as the SlowFast model or the TimeSformer model for processing video segments.

[0119] In one alternative implementation, the second type of second model can be one or more, such as the CLIP model or the ConvNext model that processes video frames.

[0120] In this embodiment, the server can acquire multiple video samples. Optionally, each video sample can be labeled with quality annotation information. Optionally, the quality annotation information can be implemented using a score of "1 to 5". In an optional embodiment, the score of "1 to 5" can be divided into multiple categories of quality annotation information.

[0121] For example, the quality labeling information in the "1-2" range is the first category of quality labeling information, the quality labeling information in the "2-3" range is the second category of quality labeling information, the quality labeling information in the "3-3.5" range is the third category of quality labeling information, the quality labeling information in the "3.5-4" range is the fourth category of quality labeling information, and the quality labeling information in the "4-5" range is the fifth category of quality labeling information.

[0122] Among them, the first category of quality annotation information indicates that the content of the corresponding video sample is no longer identifiable; the second category of quality annotation information indicates that the main target in the corresponding video sample can be observed, but the quality is obviously degraded, which may be manifested in the form of blur or noise; the third category of quality annotation information indicates that the main target in the corresponding video sample can be clearly seen, but there is obvious noise or the image is slightly blurry, or there is a small area of ​​glare, sharpened image edges, etc.; the fourth category of quality annotation information indicates that the subject of the corresponding video sample is clear and sharp, the noise is not obvious, the image has no glare, and there is no obvious jitter or blur; the fifth category of quality annotation information indicates that the subject of the corresponding video sample is clear and sharp, with no glare, no noise, and no jitter or blur.

[0123] In this embodiment, the quality annotation information carried by each video sample in the multiple video samples can be manually annotated. In one implementation of obtaining multiple video samples with quality annotation information, the server can obtain the quality annotation information set corresponding to each video sample in the multiple video samples. The number of quality annotations in the quality annotation information set corresponding to each video sample is the same; for example, the quality annotation information set corresponding to each video sample includes 100 quality annotations. Different quality annotations in the quality annotation information set corresponding to each video sample can be annotated by different individuals, that is, 100 individuals annotate each video sample in the multiple video samples respectively, obtaining the quality annotation information set corresponding to each video sample.

[0124] In one alternative embodiment, the server can directly determine the quality labeling information of each video sample based on the mean of the quality labeling information in the quality labeling information set for each video sample.

[0125] In another alternative embodiment, the multiple video samples may include two video samples derived from the same video sample, such as video sample A and video sample B, which are both cropped from the same video sample C. The quality annotation information of video sample A and video sample B should theoretically be consistent when manually annotated by the same person, due to the different evaluation standards.

[0126] Therefore, when the server obtains the quality annotation information for each video sample provided manually, it can determine the video sample group with the same source identifier based on the source identifier of each video sample (e.g., the source identifier for video sample A and video sample B is video sample C). If the quality annotation information of the video samples in the video sample group with the same source identifier is the same, then the manual annotation is determined to be qualified; if the quality annotation information of the video samples in the video sample group with the same source identifier is different, then the manual annotation is determined to be unqualified. Therefore, if the server determines that there are unqualified annotations, it can delete the quality annotation information of each video sample corresponding to the unqualified manual annotation, obtaining a set of quality annotation information for each video sample in multiple video samples. Subsequently, the server can directly determine the quality annotation information of each video sample based on the average of the quality annotation information in each video sample's quality annotation information set.

[0127] The above embodiments ensure that the quality labeling standards for multiple video samples are the same, thereby providing more impartial quality labeling information.

[0128] In one optional embodiment, if the number of first models is one and the number of second models is one, the server can perform feature extraction on multiple video samples based on the first model to obtain first feature information for each video sample, and perform feature extraction on multiple video samples based on the second model to obtain second feature information for each video sample.

[0129] In another optional embodiment, if there are multiple first models and multiple second models, the server can perform feature extraction on multiple video samples based on each first model to obtain the first feature information of each video sample corresponding to each first model, and perform feature extraction on multiple video samples based on each second model to obtain the second feature information of each video sample corresponding to each second model.

[0130] In another alternative embodiment, the number of first models is multiple and the number of second models is one; or the number of first models is one and the number of second models is multiple.

[0131] Figure 3 This is a flowchart illustrating the determination of first feature information according to an exemplary embodiment, such as... Figure 3 As shown, the process includes the following steps S301-S305:

[0132] In step S301, frame extraction is performed on each of the multiple video samples to obtain a preset number of image frames for each video sample.

[0133] In this embodiment of the application, in order to save feature extraction time and improve feature extraction efficiency, before performing feature extraction on each video sample, the server can perform frame extraction on each of the multiple video samples to obtain a preset number of image frames for each video sample.

[0134] Optionally, when extracting frames from each video sample, the server can first determine the number of frames to be extracted, and then determine the extraction interval based on the total number of frames in each video sample and the number of frames to be extracted. Alternatively, the server can obtain a preset extraction interval and then extract frames from each video sample to obtain a pre-defined number of image frames. In this latter case, the number of image frames extracted from each video sample is different.

[0135] In step S303, a preset number of image frames in each video sample are sorted according to their sequential positions within the video sample to obtain a video segment for each video sample.

[0136] Optionally, the server can sort a preset number of image frames in each video sample according to the order of their positions in the video sample, so that the positional relationship of the image frames in the video segment of each video sample is consistent with their positional relationship in the video sample. In this way, the video segment of each video sample can contain temporal information, which can then be included when extracting the first feature information.

[0137] In step S305, feature extraction is performed on the video segments of each video sample based on the first model to obtain the first feature information of each video sample.

[0138] Optionally, the server can treat the video segments of each video sample as a whole and use the first model to extract the overall features of the video sample to obtain the first feature information of each video sample.

[0139] Optionally, the server can also treat each video sample as a whole and use the first model to extract its overall features to obtain the first feature information of each video sample.

[0140] Figure 4 This is a flowchart illustrating the determination of second feature information according to an exemplary embodiment, such as... Figure 4 As shown, the process includes the following steps S401-S405:

[0141] In step S401, frame extraction is performed on each of the multiple video samples to obtain a preset number of image frames for each video sample.

[0142] In this embodiment of the application, in order to save feature extraction time and improve feature extraction efficiency, before performing feature extraction on each video sample, the server can perform frame extraction on each of the multiple video samples to obtain a preset number of image frames for each video sample.

[0143] Optionally, when the server extracts frames for each video sample, it can first determine the number of frames to be extracted, and then determine the extraction interval based on the total number of frames in each video sample and the number of frames to be extracted. Alternatively, the server can obtain a preset extraction interval and then extract frames for each video sample to obtain a pre-set number of image frames. In this case, the number of image frames extracted for each video sample is different. In step S403, feature extraction is performed on each image frame in the preset number of image frames for each video sample based on the second model to obtain sub-feature information of each image frame in the preset number of image frames for each video sample.

[0144] Optionally, the server can perform feature extraction on each image frame of a preset number of image frames for each video sample based on the second model, obtaining sub-feature information for each image frame of the preset number of image frames for each video sample. In other words, the second model can perform frame-by-frame feature extraction on the preset number of image frames.

[0145] In another alternative embodiment, the server can perform feature extraction on each image frame in each video sample based on the second model to obtain sub-feature information of each image frame in each video sample.

[0146] In step S405, the second feature information of each video sample is determined based on the sub-feature information of each image frame in a preset number of image frames for each video sample.

[0147] In this embodiment, the server can determine the second feature information of each video sample based on the mean of the sub-feature information of each image frame in a preset number of image frames of each video sample.

[0148] Alternatively, in this embodiment of the application, the server can determine the second feature information of each video sample based on the mean of the sub-feature information of each image frame in each video sample.

[0149] In step S203, based on the feature dimension transformation module in the quality assessment model, feature dimension transformation and feature information extraction are performed on the first feature information and the second feature information of each video sample to determine the comprehensive feature information of each video sample.

[0150] In one optional embodiment, if the dimensions of the first feature information and the second feature information of each video sample are the same, the server can aggregate the first feature information and the second feature information of each video sample to obtain the comprehensive feature information of each video sample.

[0151] In another optional embodiment, if the dimensions of the first feature information and the second feature information of each video sample are different, the server can perform feature dimension transformation and feature information extraction on the first feature information and the second feature information of each video sample to determine the comprehensive feature information of each video sample.

[0152] Optionally, in this embodiment of the application, the feature dimension transformation module in the quality assessment model can be used to perform feature dimension transformation and feature information extraction on the first feature information and the second feature information of each video sample to determine the comprehensive feature information of each video sample.

[0153] Figure 5 This is a flowchart illustrating the determination of comprehensive feature information according to an exemplary embodiment, such as... Figure 5 As shown, the process includes the following steps S501-S503:

[0154] In step S501, the feature dimension transformation module is used to perform feature dimension transformation and feature information extraction on the first feature information and the second feature information of each video sample, respectively, to obtain the first feature information and the second feature information with the same feature dimension for each video sample.

[0155] In one optional embodiment, the quality assessment model may include a feature dimension transformation module. The server can use this module to transform the first feature information and the second feature information of each video sample, respectively, to obtain first and second feature information with the same feature dimensions for each video sample. For example, assuming the first feature information of video sample A contains 200 dimensions and the second feature information contains 300 dimensions, the server can use the feature dimension transformation module to transform the first feature information of video sample A (i.e., feature extraction) to obtain 100-dimensional first feature information, and then use the same module to transform the second feature information of video sample A (i.e., feature extraction) to obtain 100-dimensional second feature information.

[0156] In step S503, the first feature information and the second feature information with the same feature dimension for each video sample are aggregated to obtain the comprehensive feature information of each video sample.

[0157] Subsequently, the server can aggregate the first and second feature information with the same feature dimension for each video sample to obtain the comprehensive feature information for each video sample. For example, the server can add the first and second feature information with the same feature dimension for each video sample to obtain the comprehensive feature information for each video sample. Optionally, the dimension of the comprehensive feature information can be 200.

[0158] In step S205, the quality assessment model is trained using the intra-sample consistency loss function, the inter-sample difference loss function, and the sample true label difference loss function, as well as the first feature information, second feature information, comprehensive feature information, and quality annotation information of each video sample, to obtain the trained quality assessment model. The intra-sample consistency loss function is used to constrain the feature information of the same video sample under different types of models to be in a consistent feature latent space. The inter-sample difference loss function is used to constrain the feature information of video samples carrying different quality annotation information under the same model to be in different feature latent spaces.

[0159] Figure 6This is a flowchart illustrating a method for determining a trained quality assessment model according to an exemplary embodiment, such as... Figure 6 As shown, the process includes the following steps S601-S611:

[0160] In step S601, the target feature information is obtained by weighting each sub-feature information based on the weight data corresponding to each sub-feature information in the comprehensive feature information generated by the weight allocation module.

[0161] In this embodiment of the application, the quality assessment model also includes a weight allocation module. The weight allocation module is set after the feature dimension transformation module, so that the comprehensive feature information composed of the first feature information and the second feature information can be input into the weight allocation module.

[0162] In an optional embodiment, the first feature information and the second feature information in the comprehensive feature information can be considered as sub-feature information in the comprehensive feature information.

[0163] Optionally, the feature dimension transformation module consists of two fully connected networks. Each fully connected network can be followed by a normalization layer and an activation layer. After dimension transformation, the first feature information, second feature information, and other features processed by the weight allocation module are weighted and summed to obtain a unified feature representation.

[0164]

[0165] Among them, h (i) Indicates the comprehensive feature information of each video sample; N indicates the number of models (including the first model and the second model); Indicates the feature information corresponding to each model; ω n Indicates the weights corresponding to each model.

[0166] The weighted composite feature information is used as the final quality information representation, which is then input into the regression head to predict video quality.

[0167] In step S603, the target feature information of each video sample is predicted using the sample true label difference loss function to obtain the quality prediction information of each video sample.

[0168] Optionally, the server can use the sample true label difference loss function to input the target feature information of each video sample into the regression head for quality prediction, thereby obtaining the quality prediction information for each video sample. The sample true label difference loss function refers to either the L1 loss function (i.e., absolute average error) or the L2 loss function (i.e., mean squared error).

[0169] In step S605, the quality loss data of each video sample is determined based on the quality prediction information and quality identification information of each video sample.

[0170] In this embodiment, the server can determine the quality loss data of each video sample based on the quality prediction information and quality identification information of each video sample.

[0171] In step S607, the first feature information and the second feature information of each video sample are processed using the intra-sample consistency loss function to determine the intra-sample loss data of each video sample.

[0172] The above method makes the entire training process very efficient, but due to the limited number of learnable parameters, the following problems will be encountered:

[0173] First, since the upstream tasks (i.e. the original tasks) of the model have significant differences, the feature information they extract may not have a consistent representation. It is very important to constrain this feature information to the representation space of video quality.

[0174] Second, unlike video classification, which has objective metrics, video quality assessment is a highly subjective task, correlated with multiple factors (such as content, loss type and degree, motion patterns and intensity). Videos with the same content may have different quality, and videos with different content may have the same quality. Therefore, pre-trained models based on objective metrics are difficult to distinguish from the outliers mentioned above.

[0175] This application proposes to use the Intra-Consistency Loss function to constrain the feature information extracted by different models corresponding to the same video sample, so that they are in a consistent feature latent space.

[0176] In one optional embodiment, the server can determine the intra-sample loss data of each video sample based on the first feature information and the second feature information of each video sample; or, if there are multiple first models, the server can determine the intra-sample loss data of each video sample based on two different first feature information of each video sample; or, if there are multiple second models, the server can determine the intra-sample loss data of each video sample based on two different second feature information of each video sample; or, the server can determine the intra-sample loss data of each video sample based on every two feature information from the first and second feature information contained in each video sample, as shown in the following formula:

[0177]

[0178] Among them, f n and fm This indicates the feature information output by different models for the same video sample.

[0179] In step S609, the first feature information and the second feature information of each video sample are processed using the inter-sample difference loss function to determine the inter-sample loss data.

[0180] This application proposes using the Inter-Divisibility Loss function to bring the feature information of video samples belonging to the same quality labeling information closer together, and to push the feature information of video samples belonging to different quality labeling information further apart.

[0181] In this embodiment, the server can use the inter-sample discrepancy loss function to process two first feature information of any two video samples carrying different quality annotation information output by the first model to determine the first inter-sample loss data. Alternatively, the server can use the inter-sample discrepancy loss function to process two second feature information of any two video samples carrying different quality annotation information output by the second model to determine the first inter-sample loss data. Alternatively, the server can use the inter-sample discrepancy loss function to process the first feature information of a first video sample and the second feature information of a second video sample carrying different quality annotation information to determine the first inter-sample loss data.

[0182] In this embodiment, the server can use the inter-sample discrepancy loss function to process two first feature information of any two video samples carrying the same quality annotation information output by the first model to determine the second inter-sample loss data. Alternatively, the server can use the inter-sample discrepancy loss function to process two second feature information of any two video samples carrying the same quality annotation information output by the second model to determine the second inter-sample loss data. Alternatively, the server can use the inter-sample discrepancy loss function to process the first feature information of the first video sample and the second feature information of the second video sample carrying the same quality annotation information to determine the second inter-sample loss data.

[0183] In this embodiment, the server can determine the inter-sample loss data based on the first inter-sample loss data and the second inter-sample loss data. The specific formula is as follows:

[0184]

[0185]

[0186] Among them, c k and c t S represents the cluster centers, where k and t are distinct centers; t and Sk This represents a cluster.

[0187] In step S611, the quality assessment model is trained using quality loss data, intra-sample loss data, and inter-sample loss data to obtain the trained quality assessment model.

[0188] In this embodiment, the server can train the quality assessment model using quality loss data, intra-sample loss data, and inter-sample loss data to obtain the trained quality assessment model, as shown in the following formula:

[0189]

[0190] Specifically, the quality assessment model can be trained using quality loss data, intra-sample loss data, and between-sample loss data. Training can be stopped after the following three conditions are met, yielding the trained quality assessment model. These three conditions can be:

[0191] First, until, for the same video sample, the first similarity data between the first feature information output by the first model and the second feature information output by the second model in the quality assessment model is less than the first preset similarity data; or, until, for the same video sample, the first similarity data between the first feature information output by any two first models in the quality assessment model is less than the first preset similarity data; or, until, for the same video sample, the first similarity data between the second feature information output by any two second models in the quality assessment model is less than the first preset similarity data.

[0192] Second, the feature information of video samples with different quality annotations is pushed further apart until, for two video samples with different quality annotations, the second similarity data between the two first feature information output by the first model in the quality assessment model is greater than the second preset similarity data; or, for two video samples with different quality annotations, the second similarity data between the two second feature information output by the second model in the quality assessment model is greater than the second preset similarity data. And, for two video samples with the same quality annotations, the third similarity data between the two second feature information output by the second model in the trained quality assessment model is less than the third preset similarity data; or, for two video samples with the same quality annotations, the third similarity data between the two first feature information output by the first model in the trained quality assessment model is less than the third preset similarity data. This is to bring the feature information of video samples with the same quality annotations closer together.

[0193] Third, the gap between the predicted annotation information and the quality annotation information of each video sample is kept within a certain tolerance range until the fourth similarity data between the predicted annotation information determined based on the comprehensive feature information of each video sample and the quality annotation information of each video sample is less than the fourth preset similarity data.

[0194] In this way, the feature information of the same video sample can be represented as consistently as possible in different models. At the same time, if different video samples have different video quality, their feature information can be pushed further apart.

[0195] Therefore, after the quality assessment model is trained, it can be tested on test videos. If, for the same test video, the first similarity data between the first feature information output by the first model and the second feature information output by the second model in the trained quality assessment model is less than the first preset similarity data, and for two test videos with different quality labeling information, the second similarity data between the two first feature information output by the first model in the trained quality assessment model is greater than the second preset similarity data, and for two test videos with the same quality labeling information, the third similarity data between the two second feature information output by the second model in the trained quality assessment model is less than the third preset similarity data, and the third preset similarity data is less than the second preset similarity data, then the quality assessment model can be determined to be usable.

[0196] The multiple video samples can include test videos, meaning that previously used video samples can be used for testing. Alternatively, new test videos can be used to evaluate the model.

[0197] Figure 7 This is a flowchart illustrating a video quality determination method according to an exemplary embodiment, such as... Figure 7 As shown, the video quality assessment model training method can be applied to servers or other node devices, such as clients. The following explanation uses a server as an example. The method includes at least the following steps S701 to S703:

[0198] In step S701, the video to be determined is identified.

[0199] In this embodiment of the application, after the quality assessment model is trained, it can be put into application. First, the server can determine the video whose quality needs to be determined.

[0200] In step S703, feature extraction is performed on the video to be determined based on the first model to obtain the first feature information of the video to be determined.

[0201] Optionally, if there is only one first model, the server can extract features from the video to be determined based on that first model to obtain the first feature information of the video to be determined. If there are multiple first models, the server can use each first model to extract features from the video to be determined to obtain multiple first feature information.

[0202] In step S705, feature extraction is performed on the video to be determined based on the second model to obtain the second feature information of the video to be determined.

[0203] Optionally, if there is only one first model, the server can extract features from the video to be determined based on that second model to obtain second feature information of the video to be determined. If there are multiple first models, the server can use each second model to extract features from the video to be determined to obtain multiple pieces of second feature information.

[0204] In step S707, the first feature information and the second feature information of the video to be determined are input into the trained quality assessment model to obtain the quality data of the video to be determined.

[0205] Optionally, the server can input the first and second feature information of the video to be determined into the trained quality assessment model. The feature dimension transformation module in the quality assessment model then performs feature dimension transformation and feature extraction on the first and second feature information to obtain first and second feature information with consistent dimensions. Subsequently, the weight allocation module assigns weights to the first and second feature information, ensuring that the dependency weights on different feature information are different during later quality predictions. This can be achieved through self-learning.

[0206] Figure 8 This is a block diagram illustrating a video quality assessment model training apparatus according to an exemplary embodiment. The apparatus has the function of implementing the data processing method in the above-described method embodiments; the function can be implemented in hardware or by hardware executing corresponding software. (Refer to...) Figure 8 The device includes a feature extraction module 801, a feature determination module 801, and a model training module 803.

[0207] The feature extraction module 801 is configured to perform feature extraction on multiple video samples based on a first model to obtain first feature information for each video sample; and to perform feature extraction on multiple video samples based on a second model to obtain second feature information for each video sample; each video sample carries quality annotation information; the first feature information is the information corresponding to the video segment; and the second feature information is the information corresponding to the video frame.

[0208] The feature determination module 802 is configured to perform feature dimension transformation and feature information extraction on the first feature information and the second feature information of each video sample based on the feature dimension transformation module in the quality assessment model, and determine the comprehensive feature information of each video sample.

[0209] The model training module 803 is configured to train the quality assessment model using the intra-sample consistency loss function, the inter-sample difference loss function, and the sample true label difference loss function, as well as the first feature information, second feature information, comprehensive feature information, and quality annotation information of each video sample, to obtain the trained quality assessment model.

[0210] The intra-sample consistency loss function is used to constrain the feature information of the same video sample under different types of models to be in a consistent feature latent space; the inter-sample difference loss function is used to constrain the feature information of video samples carrying different quality annotation information to be in different feature latent spaces under the same model.

[0211] In some possible embodiments, the feature determination module is configured to perform:

[0212] The feature dimension transformation module is used to transform the first feature information and extract the feature information of each video sample, respectively, to obtain the first feature information and the second feature information of each video sample with the same feature dimension.

[0213] The first and second feature information with the same feature dimension for each video sample are aggregated to obtain the comprehensive feature information for each video sample.

[0214] In some possible embodiments, the quality assessment model also includes a weight allocation module;

[0215] The model training module is configured to execute:

[0216] Based on the weight allocation module, the weight data corresponding to each sub-feature information in the comprehensive feature information is used to weight each sub-feature information to obtain the target feature information;

[0217] The quality prediction information of each video sample is obtained by using the sample true label difference loss function to predict the target feature information of each video sample.

[0218] The quality loss data for each video sample is determined based on the quality prediction information and quality identification information of each video sample.

[0219] The first and second feature information of each video sample are processed using the intra-sample consistency loss function to determine the intra-sample loss data for each video sample.

[0220] The first and second feature information of each video sample are processed using the inter-sample difference loss function to determine the inter-sample loss data.

[0221] The quality assessment model is trained using quality loss data, intra-sample loss data, and inter-sample loss data to obtain the trained quality assessment model.

[0222] In some possible embodiments, the model training module is configured to perform:

[0223] The first inter-sample loss data is determined by processing two first feature information of any two video samples carrying different quality annotation information in the output of the first model using the inter-sample difference loss function; or, the first inter-sample loss data is determined by processing two second feature information of any two video samples carrying different quality annotation information in the output of the second model using the inter-sample difference loss function.

[0224] The inter-sample loss function is used to process two first feature information of any two video samples carrying the same quality annotation information output by the first model to determine the second inter-sample loss data; or, the inter-sample loss function is used to process two second feature information of any two video samples carrying the same quality annotation information output by the second model to determine the second inter-sample loss data.

[0225] The inter-sample loss data is determined based on the first inter-sample loss data and the second inter-sample loss data.

[0226] In some possible embodiments, the model training module is configured to perform:

[0227] The quality assessment model is trained using quality loss data, intra-sample loss data, and inter-sample loss data until, for the same video sample, the first similarity data between the first feature information output by the first model and the second feature information output by the second model in the quality assessment model is less than the first preset similarity data; until, for two video samples carrying different quality annotation information, the second similarity data between the two first feature information output by the first model in the quality assessment model is greater than the second preset similarity data; until, for two video samples carrying the same quality annotation information, the third similarity data between the two second feature information output by the second model in the trained quality assessment model is less than the third preset similarity data; the third preset similarity data is less than the second preset similarity data; and until, the fourth similarity data between the predicted annotation information determined based on the comprehensive feature information of each video sample and the quality annotation information of each video sample is less than the fourth preset similarity data, then the model training stops, and the trained quality assessment model is obtained.

[0228] In some possible embodiments, the apparatus further includes a model determination module configured to perform:

[0229] Multiple models are identified; each model is used to process at least one original task; the feature representations obtained by each model in processing the original task are related to video quality.

[0230] Among multiple models, the model related to video segment processing is identified as the first model;

[0231] Among the multiple models, the one related to video frame processing is identified as the second model.

[0232] In some possible embodiments, the feature extraction module is configured to perform:

[0233] Frame extraction is performed on each video sample from multiple video samples to obtain a preset number of image frames for each video sample.

[0234] The image frames of each video sample are sorted according to their sequential position within the video sample to obtain a video segment for each video sample.

[0235] Based on the first model, feature extraction is performed on the video segments of each video sample to obtain the first feature information of each video sample.

[0236] In some possible embodiments, the feature extraction module is configured to perform:

[0237] Frame extraction is performed on each video sample from multiple video samples to obtain a preset number of image frames for each video sample.

[0238] Based on the second model, feature extraction is performed on each image frame in a preset number of image frames for each video sample to obtain sub-feature information of each image frame in a preset number of image frames for each video sample.

[0239] The second feature information of each video sample is determined based on the sub-feature information of each image frame in a preset number of image frames for each video sample.

[0240] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0241] Figure 9 This is a block diagram illustrating a video quality determination apparatus according to an exemplary embodiment. The apparatus has the function of implementing the data processing method in the above-described method embodiments; the function can be implemented in hardware or by hardware executing corresponding software. (Refer to...) Figure 9 The device includes a video determination module 901, a first extraction module 902, a second extraction module 903, and a quality determination module 904.

[0242] The video determination module 901 is configured to determine the video to be determined.

[0243] The first extraction module 902 is configured to perform feature extraction on the video to be determined based on the first model, and obtain the first feature information of the video to be determined.

[0244] The second extraction module 903 is configured to perform feature extraction on the video to be determined based on the second model, and obtain the second feature information of the video to be determined.

[0245] The quality determination module 904 is configured to input the first and second feature information of the video to be determined into the trained quality assessment model to obtain the quality data of the video to be determined.

[0246] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0247] Figure 10 This is a block diagram illustrating an apparatus 3000 for training a video quality assessment model or determining video quality according to an exemplary embodiment. For example, apparatus 3000 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0248] Reference Figure 10 The device 3000 may include one or more of the following components: a processing component 3002, a memory 3004, a power component 3006, a multimedia component 3008, an audio component 3010, an input / output (I / O) interface 3012, a sensor component 3014, and a communication component 3016.

[0249] Processing component 3002 typically controls the overall operation of device 3000, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 3002 may include one or more processors 3020 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 3002 may include one or more modules to facilitate interaction between processing component 3002 and other components. For example, processing component 3002 may include a multimedia module to facilitate interaction between multimedia component 3008 and processing component 3002.

[0250] Memory 3004 is configured to store various types of data to support the operation of device 3000. Examples of this data include instructions for any application or method operating on device 3000, contact data, phonebook data, messages, pictures, videos, etc. Memory 3004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0251] Power supply component 3006 provides power to various components of device 3000. Power supply component 3006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 3000.

[0252] Multimedia component 3008 includes a screen that provides an output interface between device 3000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 3008 includes a front-facing camera and / or a rear-facing camera. When device 3000 is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0253] Audio component 3010 is configured to output and / or input audio signals. For example, audio component 3010 includes a microphone (MIC) configured to receive external audio signals when device 3000 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 3004 or transmitted via communication component 3016. In some embodiments, audio component 3010 also includes a speaker for outputting audio signals.

[0254] I / O interface 3012 provides an interface between processing component 3002 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0255] Sensor assembly 3014 includes one or more sensors for providing state assessments of various aspects of device 3000. For example, sensor assembly 3014 may detect the on / off state of device 3000, the relative positioning of components such as the display and keypad of device 3000, changes in the position of device 3000 or a component of device 3000, the presence or absence of user contact with device 3000, the orientation or acceleration / deceleration of device 3000, and temperature changes of device 3000. Sensor assembly 3014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 3014 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 3014 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0256] Communication component 3016 is configured to facilitate wired or wireless communication between device 3000 and other devices. Device 3000 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 3016 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 3016 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0257] In an exemplary embodiment, the apparatus 3000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0258] Embodiments of the present invention also provide a computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a video quality assessment model training method. The at least one instruction or the at least one program is loaded and executed by the processor to implement the video quality assessment model training method provided in the above-described method embodiments.

[0259] Embodiments of the present invention also provide a computer program product comprising a computer program stored in a readable storage medium, wherein at least one processor of a computer device reads from the readable storage medium and executes the computer program, causing the computer device to perform the method of any of the first aspects of the embodiments of the present disclosure.

[0260] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0261] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0262] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0263] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for training a video quality assessment model, characterized in that, include: Based on the first model, feature extraction is performed on multiple video samples to obtain the first feature information of each video sample; Based on the second model, feature extraction is performed on the multiple video samples to obtain the second feature information of each video sample; each video sample carries quality annotation information; the first feature information is the information corresponding to the video segment; the second feature information is the information corresponding to the video frame; Based on the feature dimension transformation module in the quality assessment model, feature dimension transformation and feature information extraction are performed on the first feature information and the second feature information of each video sample to determine the comprehensive feature information of each video sample; The quality assessment model is trained using the intra-sample consistency loss function, the inter-sample difference loss function, and the sample true label difference loss function, as well as the first feature information, second feature information, comprehensive feature information, and quality annotation information of each video sample, to obtain the trained quality assessment model. The intra-sample consistency loss function is used to constrain the feature information of the same video sample under different types of models to be in a consistent feature latent space; the inter-sample difference loss function is used to constrain the feature information of video samples carrying different quality annotation information under the same model to be in different feature latent spaces.

2. The video quality assessment model training method according to claim 1, characterized in that, The feature dimension transformation module based on the quality assessment model performs feature dimension transformation and feature information extraction on the first feature information and the second feature information of each video sample to determine the comprehensive feature information of each video sample, including: The feature dimension transformation module is used to perform feature dimension transformation and feature information extraction on the first feature information and the second feature information of each video sample, respectively, to obtain the first feature information and the second feature information with the same feature dimension for each video sample; The first and second feature information with the same feature dimension for each video sample are aggregated to obtain the comprehensive feature information for each video sample.

3. The video quality assessment model training method according to claim 1, characterized in that, The quality assessment model also includes a weight allocation module; The quality assessment model is trained using the intra-sample consistency loss function, the inter-sample difference loss function, and the sample true label difference loss function, as well as the first feature information, second feature information, comprehensive feature information, and quality annotation information of each video sample, to obtain the trained quality assessment model, including: Based on the weight allocation module, the weight data corresponding to each sub-feature information in the comprehensive feature information is used to weight the sub-feature information to obtain the target feature information. The target feature information of each video sample is used to predict the quality of the video sample by using the sample true label difference loss function; The quality loss data of each video sample is determined based on the quality prediction information and quality identification information of each video sample; The first feature information and the second feature information of each video sample are processed using the intra-sample consistency loss function to determine the intra-sample loss data of each video sample. The first and second feature information of each video sample are processed using the inter-sample difference loss function to determine the inter-sample loss data. The quality assessment model is trained using the quality loss data, the intra-sample loss data, and the inter-sample loss data to obtain the trained quality assessment model.

4. The video quality assessment model training method according to claim 3, characterized in that, The process of using the inter-sample difference loss function to process the first and second feature information of each video sample to determine the inter-sample loss data includes: The first inter-sample loss data is determined by processing two first feature information of any two video samples carrying different quality labeling information output by the first model using the inter-sample difference loss function; or, the first inter-sample loss data is determined by processing two second feature information of any two video samples carrying different quality labeling information output by the second model using the inter-sample difference loss function. The inter-sample difference loss function is used to process two first feature information of any two video samples carrying the same quality labeling information output by the first model to determine the second inter-sample loss data; or, the inter-sample difference loss function is used to process two second feature information of any two video samples carrying the same quality labeling information output by the second model to determine the second inter-sample loss data. The inter-sample loss data is determined based on the first inter-sample loss data and the second inter-sample loss data.

5. The video quality assessment model training method according to claim 3, characterized in that, The step of training the quality assessment model using the quality loss data, the intra-sample loss data, and the inter-sample loss data to obtain the trained quality assessment model includes: The quality assessment model is trained using the quality loss data, the intra-sample loss data, and the inter-sample loss data until, for the same video sample, the first similarity data between the first feature information output by the first model and the second feature information output by the second model in the quality assessment model is less than a first preset similarity data; until, for two video samples carrying different quality labeling information, the second similarity data between the two first feature information output by the first model in the quality assessment model is greater than a second preset similarity data; until, for two video samples carrying the same quality labeling information, the third similarity data between the two second feature information output by the second model in the trained quality assessment model is less than a third preset similarity data; the third preset similarity data is less than the second preset similarity data; and until, the fourth similarity data between the predicted labeling information determined based on the comprehensive feature information of each video sample and the quality labeling information of each video sample is less than a fourth preset similarity data, the model training is stopped, and the trained quality assessment model is obtained.

6. The video quality assessment model training method according to any one of claims 1-5, characterized in that, The method further includes: Multiple models are identified; each of the multiple models is used to process at least one original task; the feature representations obtained by each model in processing the original task are related to video quality; The model related to video segment processing among the multiple models is identified as the first model; The model related to video frame processing among the multiple models is identified as the second model.

7. The video quality assessment model training method according to any one of claims 1-5, characterized in that, The step of extracting features from multiple video samples based on the first model to obtain the first feature information of each video sample includes: Frame extraction is performed on each of the plurality of video samples to obtain a preset number of image frames for each video sample; The image frames of each video sample are sorted according to their sequential positions within the video sample to obtain a video segment for each video sample. Based on the first model, feature extraction is performed on the video segments of each video sample to obtain the first feature information of each video sample.

8. The video quality assessment model training method according to any one of claims 1-5, characterized in that, The step of extracting features from the multiple video samples based on the second model to obtain the second feature information of each video sample includes: Frame extraction is performed on each of the plurality of video samples to obtain a preset number of image frames for each video sample; Based on the second model, feature extraction is performed on each image frame in a preset number of image frames for each video sample to obtain sub-feature information of each image frame in a preset number of image frames for each video sample; The second feature information of each video sample is determined based on the sub-feature information of each image frame in a preset number of image frames for each video sample.

9. A method for determining video quality, characterized in that, include: Confirm the video to be confirmed; Based on the first model, feature extraction is performed on the video to be determined to obtain the first feature information of the video to be determined; Based on the second model, feature extraction is performed on the video to be determined to obtain the second feature information of the video to be determined. The first and second feature information of the video to be determined are input into any of the trained quality assessment models in claims 1-8 to obtain the quality data of the video to be determined.

10. A video quality assessment model training device, characterized in that, include: The feature extraction module is configured to perform feature extraction on multiple video samples based on a first model to obtain the first feature information of each video sample. Based on the second model, feature extraction is performed on the multiple video samples to obtain the second feature information of each video sample; each video sample carries quality annotation information; the first feature information is the information corresponding to the video segment; the second feature information is the information corresponding to the video frame; The feature determination module is configured to perform feature dimension transformation and feature information extraction on the first feature information and the second feature information of each video sample based on the feature dimension transformation module in the quality assessment model, and determine the comprehensive feature information of each video sample; The model training module is configured to train the quality assessment model using the intra-sample consistency loss function, the inter-sample difference loss function, and the sample true label difference loss function, as well as the first feature information, second feature information, comprehensive feature information, and quality labeling information of each video sample, to obtain the trained quality assessment model. The intra-sample consistency loss function is used to constrain the feature information of the same video sample under different types of models to be in a consistent feature latent space; the inter-sample difference loss function is used to constrain the feature information of video samples carrying different quality annotation information under the same model to be in different feature latent spaces.

11. A video quality determination device, characterized in that, include: The video determination module is configured to determine the video to be determined. The first extraction module is configured to perform feature extraction on the video to be determined based on a first model to obtain the first feature information of the video to be determined. The second extraction module is configured to perform feature extraction on the video to be determined based on the second model to obtain the second feature information of the video to be determined. The quality determination module is configured to input the first feature information and the second feature information of the video to be determined into any one of the trained quality assessment models in claims 1-8 to obtain the quality data of the video to be determined.

12. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the video quality assessment model training method as described in any one of claims 1 to 8 or the video quality determination method as described in claim 9.

13. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the video quality assessment model training method as described in any one of claims 1 to 8 or the video quality determination method as described in claim 9.

Images