Methods, systems, and media for determining a perceptual quality indicator for a video content item

Abstract

Methods, systems, and media for determining a perceptual quality indicator of a video content item are provided. In some embodiments, the method includes receiving a video content item, extracting a plurality of frames from the video content item, determining a content quality indicator for each frame of the plurality of frames of the video content item using a first subnetwork of a deep neural network, determining a video distortion indicator for each frame of the plurality of frames of the video content item using a second subnetwork of the deep neural network, determining a compression sensitivity indicator for each frame of the plurality of frames of the video content item using a third subnetwork of the deep neural network, generating a quality level for each frame of the plurality of frames of the video content item, the quality level linking the content quality indicator, the video distortion indicator, and the compression sensitivity indicator for the frame of the video content item, generating an overall quality level for the video content item by aggregating the quality level for each frame of the plurality of frames, and causing a video recommendation to be presented based on the overall quality level for the video content item.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 210,003, filed June 12, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The disclosed topics relate to methods, systems, and media for determining perceptual quality metrics for video content items. Background Technology

[0004] Users frequently stream video content items (e.g., TV shows, videos, movies, music videos, etc.) from servers hosting the video content items on their devices. Service providers typically optimize and stream transcoded versions of such video content items. However, this includes both professionally created content and user-generated content. In professionally created content, the service provider receives the original, unprocessed version of the video content item; in user-generated content, the service provider receives a non-original version of the video content item with pre-existing distortion or compression artifacts.

[0005] Understanding the perceived subjective quality of such content is a difficult and time-consuming task. For example, existing methods for determining video quality involve employing a team of individuals to manually review the video and identify visible quality issues.

[0006] Therefore, there is a need to provide new methods, systems, and media for determining the perceived quality metrics of video content items. Summary of the Invention

[0007] Methods, systems, and media are provided for determining perceptual quality metrics for video content items.

[0008] According to some embodiments of the disclosed subject matter, a method for video quality assessment is provided, the method comprising: receiving a video content item; extracting a plurality of frames from the video content item; determining a content quality metric for each frame of the plurality of frames of the video content item using a first subnetwork of a deep neural network; determining a video distortion metric for each frame of the plurality of frames of the video content item using a second subnetwork of the deep neural network; determining a compression sensitivity metric for each frame of the plurality of frames of the video content item using a third subnetwork of the deep neural network; generating a quality level for each frame of the plurality of frames of the video content item, the quality level concatenating the content quality metric, the video distortion metric, and the compression sensitivity metric for that frame of the video content item; generating an overall quality level for the video content item by aggregating the quality levels of each frame of the plurality of frames; and causing a video recommendation to be presented based on the overall quality level of the video content item.

[0009] In some embodiments, the content quality metric represents the semantic level embedding of each of the plurality of frames of the video content item, and the first subnet of the deep neural network further outputs a predicted content label, the predicted content label describing the content appearing in each of the plurality of frames of the video content item.

[0010] In some embodiments, the video distortion metric represents the distortion-sensitive embedding of each of the plurality of frames of the video content item, and the second subnetwork of the deep neural network further outputs a detected distortion type, the detected distortion type describing the distortion detected in each of the plurality of frames of the video content item.

[0011] In some embodiments, the compression sensitivity metric represents the compression-sensitive embedding of each of the plurality of frames of the video content item, and the third subnet of the deep neural network further outputs a compression level score.

[0012] In some embodiments, a convolutional neural network is used to generate the overall quality level, the convolutional neural network outputting a block score for each of the plurality of frames of the video content item and averaging the block scores.

[0013] In some embodiments, the video recommendation includes recommendations that further compress the video content items based on the overall quality level.

[0014] In some embodiments, the video recommendation includes a recommendation to the uploader of the video content item to modify a portion of the video content item based on the quality level associated with that frame of the video content item.

[0015] According to some embodiments of the disclosed subject matter, a system for video quality assessment is provided, the system comprising: a hardware processor configured to: receive a video content item; extract a plurality of frames from the video content item; determine a content quality metric for each of the plurality of frames of the video content item using a first subnetwork of a deep neural network; determine a video distortion metric for each of the plurality of frames of the video content item using a second subnetwork of the deep neural network; determine a compression sensitivity metric for each of the plurality of frames of the video content item using a third subnetwork of the deep neural network; generate a quality level for each of the plurality of frames of the video content item, the quality level concatenating the content quality metric, the video distortion metric, and the compression sensitivity metric for that frame of the video content item; generate an overall quality level for the video content item by aggregating the quality levels of each of the plurality of frames; and cause video recommendations to be presented based on the overall quality level of the video content item.

[0016] According to some embodiments of the disclosed subject matter, a non-transitory computer-readable medium is provided containing computer-executable instructions that, when executed by a processor, cause the processor to perform a method for video quality assessment, the method comprising: receiving a video content item; extracting a plurality of frames from the video content item; determining a content quality metric for each of the plurality of frames of the video content item using a first subnetwork of a deep neural network; determining a video distortion metric for each of the plurality of frames of the video content item using a second subnetwork of the deep neural network; determining a compression sensitivity metric for each of the plurality of frames of the video content item using a third subnetwork of the deep neural network; generating a quality level for each of the plurality of frames of the video content item, the quality level concatenating the content quality metric, the video distortion metric, and the compression sensitivity metric for that frame of the video content item; generating an overall quality level for the video content item by aggregating the quality levels of each of the plurality of frames; and causing a video recommendation to be presented based on the overall quality level of the video content item.

[0017] According to some embodiments of the disclosed subject matter, a system for video quality assessment is provided, the system comprising: means for receiving a video content item; means for extracting a plurality of frames from the video content item; means for determining a content quality metric for each of the plurality of frames of the video content item using a first subnetwork of a deep neural network; means for determining a video distortion metric for each of the plurality of frames of the video content item using a second subnetwork of the deep neural network; means for determining a compression sensitivity metric for each of the plurality of frames of the video content item using a third subnetwork of the deep neural network; means for generating a quality level for each of the plurality of frames of the video content item, the quality level concatenating the content quality metric, the video distortion metric, and the compression sensitivity metric for that frame of the video content item; means for generating an overall quality level of the video content item by aggregating the quality levels of each of the plurality of frames; and means for presenting video recommendations based on the overall quality level of the video content item. Attached Figure Description

[0018] The various objects, features and advantages of the disclosed subject matter can be more fully understood when considered in conjunction with the following detailed description of the disclosed subject matter, in which the same reference numerals denote the same elements.

[0019] Figure 1 and Figure 2 Illustrative examples of a process for determining a perceived quality metric for video content items, based on some embodiments of the disclosed subject matter, are shown.

[0020] Figure 3 Illustrative examples of predicted content tags representing the frame semantics of one or more frames for each video content item are shown, according to some embodiments of the disclosed subject matter.

[0021] Figure 4 Illustrative examples of predicted distortion types detected in one or more frames of each video content item, according to some embodiments of the disclosed subject matter, are shown.

[0022] Figure 5 An illustrative example is shown of the predicted compression level in one or more frames of each video content item according to some embodiments of the disclosed subject matter.

[0023] Figure 6 Illustrative examples of interfaces for perceptual quality metrics (e.g., compression level, content tag, and distortion type) of presented video content items, predicted quality scores for frames or features of each video content item, and overall quality scores for frames or features of each video content item are shown, according to some embodiments of the disclosed subject matter.

[0024] Figure 7 A schematic diagram of an illustrative system suitable for implementing the mechanism described herein for determining perceptual quality metrics of video content items, according to some embodiments of the disclosed subject matter, is shown.

[0025] Figure 8 The present invention illustrates some embodiments of the disclosed subject matter that enable... Figure 7 Detailed examples of the hardware used in the servers and / or user equipment. Detailed Implementation

[0026] According to various embodiments, mechanisms (which can include methods, systems, and media) are provided for determining perceptual quality metrics of video content items. More specifically, the mechanisms described herein are capable of decomposing the overall perceptual quality of video content items in terms of semantic content, distortion or technical quality, and compression level.

[0027] According to some embodiments, the mechanisms described herein are capable of determining the overall perceived quality of a video content item—such as a user-generated video—from multiple perspectives, such as by determining the semantic content, distortion or technical quality, and compression level of the video content item. For example, the mechanism can provide one or more input frames of a video content item as input to multiple subnetworks of a deep neural network, wherein each subnetwork is capable of extracting depth features and quality metrics from each frame of the video content item. In a more specific example, the mechanism can provide one or more input frames of a video content item to (1) a content subnetwork that predicts content labels and content quality metrics for one or more frames of the video content item, (2) a distortion subnetwork that determines the distortion type and video distortion metrics for one or more frames of the video content item, and (3) a compression subnetwork that predicts the compression level and compression sensitivity metrics for one or more frames of the video content item.

[0028] In some embodiments, features extracted from each subnet of the deep neural network can be used to provide quality metrics that are understandable or digestible to users (e.g., uploaders, content provider users, etc.). For example, a content subnet can generate predicted content tags representing the semantics of one or more input frames, such as “outdoor recreation,” “game,” “ball,” and “walking.” In another example, a distortion subnet can generate predicted distortion types representing the distortions detected in one or more input frames, such as “jitter,” “color quantization,” “lens blur,” and “denoising.” In yet another example, a compression subnet can generate compression-sensitive embedding features describing the compression levels in one or more input frames. In response to providing users with content tags and / or distortion tags as feedback, users can be given the opportunity to modify that portion of the video content item. For example, a user can understand the content quality metrics determined for that frame based on the content tags and whether those content tags are considered content of interest. In another example, a user can understand the video distortion metrics for that frame based on the types of distortion detected within that frame.

[0029] In some embodiments, the mechanism can concatenate quality metrics for each frame of a video content item and aggregate them using an aggregation subnet to obtain an overall video quality estimate for the video content item. For example, since a video content item can be a sequence of chunks or frames, where those chunks or frames can have different perceptual qualities, the mechanism can concatenate quality metrics for each frame of a video content item and aggregate the concatenated quality metrics to obtain an overall video quality estimate for the video content item.

[0030] It should be noted that the quality metrics for one or more frames and the overall video quality estimate can be used in any suitable application.

[0031] For example, this mechanism allows service providers to use an overall video quality estimate to make informed quality-of-service trade-offs (e.g., transmitting video content items at relatively high or low resolution or at relatively high or low bitrates). Continuing this example, service providers can perform adjusted bitrate streaming of video content items transmitted to user devices based on an overall video quality estimate for each video content item and / or based on quality metrics for one or more frames of each video content item.

[0032] In another example, the mechanism can use a quality metric associated with each frame of a video content item to determine that viewers of the first part of a video content item with a higher quality metric can benefit from transmission in a format with a relatively high resolution (e.g., the original content), and that viewers of the second part of a video content item with a lower quality metric can be provided in a format with a relatively low resolution that has little perceptible impact on that part of the video content item.

[0033] In yet another example, this mechanism provides service providers with an understanding of the quality of video content items, enabling them to optimize the recommendation system (e.g., if multiple videos exist in a single event) or further compress video content items with low-quality scores that have little perceptible impact on the video content items presented to viewers. Continuing with this example, the mechanism allows service providers to determine whether the format should be changed from a first format to a potential second format based on an overall video quality estimate—such as the difference in overall video quality estimates between a video content item and subsequent video content items.

[0034] In another example, the mechanism can use extracted features and quality metrics from one or more frames to provide feedback to the uploading user about a video content item. The extracted features (e.g., content labels such as "dance" and "musical ensemble," and distortion types such as "Gaussian blur" and "color saturation") can provide the uploading user with labels indicating the perceived quality of one or more frames of the video content item. Furthermore, the uploading user can modify the video content item for uploading to the content-sharing service (e.g., removing or reducing the types of distortion detected in one or more frames of the video content item to improve the overall perceived quality of the video content item).

[0035] Combination Figure 1-8 These and other features are further described in terms of the perceptual quality metrics used to determine video content items.

[0036] Go to Figure 1 According to some embodiments of the disclosed subject matter, an illustrative example 100 of a process for determining a perceived quality metric for a video content item is shown. In some embodiments, frames of process 100 can be performed by any suitable device—such as a server that hosts media content items and streams them to user devices. In some such embodiments, the server is capable of performing frames of process 100 to extract quality scores and / or quality metrics from one or more frames of the video content item and to determine an overall video quality estimate for the video content item.

[0037] Process 100 can begin at 110 by receiving an uploaded video content item. In some embodiments, process 100 can receive the video content item from any suitable source. For example, in some embodiments, the video evaluation server can receive the uploaded video from a server associated with a video sharing service that receives videos created and / or generated by users. In some embodiments, the uploaded video content item can include any suitable content (e.g., video content, corresponding audio content, subtitles, and / or any other suitable content). In some embodiments, the uploaded video can be associated with any suitable information provided by the user who uploaded the video—such as the video title, the name of the video creator, the date the video was created, the video's theme, and / or any other suitable information.

[0038] Alternatively, in some embodiments, process 100 can be initiated at 110 by receiving an indication of a video content item to be presented. In some embodiments, the indication can be received in any suitable manner. For example, in some embodiments, a link or icon representing a video can be selected from a page indicating available video content items (e.g., the homepage of a video content service, a page showing user-personalized recommendations for a user device, and / or any other suitable page). Note that in some embodiments, the indication can be received by a user device paired with a display device to be used to present the video. For example, in some embodiments, the indication can be received by a mobile phone or virtual assistant device paired with or otherwise associated with a television.

[0039] At 120, process 100 is able to extract one or more frames from the video content item. For example, in response to receiving the video content item at 110, the video content item can be separated into its video frames at 120, and each frame can be fed into a deep neural network to determine the overall perceived quality of the video content item in terms of semantic content, distortion or technical quality, and compression level.

[0040] It should be noted that each subnet of a deep neural network can receive different input frames. For example, the content subnet of a deep neural network can receive (e.g., sampled at one frame per second) extracted frames that have been resized to reduce their resolution. In another example, the distortion subnet of a deep neural network can receive frames at the native resolution of the extracted frames (e.g., sampled at one frame per second) (e.g., to avoid rescaling artifacts). In yet another example, the compression subnet of a deep neural network can receive multiple frames at the native resolution of the frames (e.g., sampled at five frames per second) (e.g., to capture both spatial and temporal artifacts).

[0041] In some embodiments, at 130, process 100 is capable of feeding one or more frames of a video content item as input to a content subnet of a deep neural network to determine one or more content tags and content quality metrics. It should be noted that process 100 is capable of sampling frames from the video content item at a rate of one frame per second, resizing the frames to a lower resolution (e.g., because the output of the content subnet should not be affected by the input resolution), and feeding the resized frames as input to the content subnet of the deep neural network. It should also be noted that the content quality metrics predicted by the content subnet of the deep neural network can encode the meaningfulness and appeal of the video content, which may influence the viewer's attention and quality sensitivity.

[0042] This is also Figure 2 As shown, frames 220 from multiple extracted frames 210 of video content items can be input into a content subnet 222, and the content subnet 222 can provide an output 224 including predicted content labels and predicted content quality metrics for frames 220.

[0043] It should be noted that the content subnet can be any suitable video classification model used to predict one or more content labels and content quality metrics for frames containing video content items. For example, a deep neural network content subnet can be a video classification model that provides semantic-level embeddings, where the video classification model can be trained on a single video frame, and where the video classification model can output content-sensitive embedding features and predicted content labels. In a more specific example, the content subnet can be customized by adding a fully connected layer with a specific size (e.g., 16, 16, 100) of output feature maps before the last layer and changing the output of the last layer to correspond to the total number of video classes (e.g., 3,862 logit values ​​to correspond to the YouTube-8M dataset with 3,862 coarse-grained and fine-grained entities). Continuing this example, it is possible to sample the input frames (e.g., at 1 frame per second) and resize them to a smaller resolution (e.g., resized to 496×496), and to fine-tune the baseline convolutional neural network in multi-label mode using cross-entropy loss. It should be noted that the input frame can have any suitable number of content tags (e.g., up to 20 content tags).

[0044] Figure 3 The document illustrates illustrative examples of predicted content tags representing the frame semantics of one or more frames for each video content item, according to some embodiments of the disclosed subject matter. Figure 3As shown, each frame 310, 320, and 330 from different video content items can be presented in the interface along with the predicted content tag and its associated probability. For example, each frame 310, 320, and 330 can be presented along with the top five predictions 312 and probabilities 314 for each frame, such as: content tag "video game" and probability 0.95, content tag "game" and probability 0.94, content tag "World of Warcraft" and probability 0.45, content tag "Warcraft" and probability 0.40, and content tag "strategy video game" and probability 0.07.

[0045] Figure 6 The document illustrates illustrative examples of frame semantics representing a video content item and predicted content tags for that frame's content quality metrics, according to some embodiments of the disclosed subject matter. For example... Figure 6 As shown, an interface is capable of presenting a representative frame 610 including video content items and predicted content labels 620 describing the content appearing within frame 610 (e.g., "dance," "musical ensemble," and "outdoor entertainment"). Also as... Figure 6 As shown, the interface can also include a Content Quality Index (CT) 630 for frame 610. As described above, the CT 630, calculated by the content subnet of the deep neural network, can encode the meaning and appeal of the video content, which may influence the viewer's attention and quality sensitivity. For example, the content subnet of the deep neural network has determined a CT of 3.621 for a representative frame 610 of video content items, indicating that frames 610 including "dance," "musical ensemble," and "outdoor entertainment" are considered interesting content.

[0046] Back Figure 1 In some embodiments, at 140, process 100 is capable of feeding one or more frames of a video content item as input to a distortion subnet of a deep neural network to determine one or more distortion types and video distortion metrics. It should be noted that video content items, such as user-generated videos, can contain various distortions orthogonal to the video content features (e.g., contrast alterations or denoising) and may have a good or bad effect on perceptual quality. For example, distortion may be introduced during the video production process, where some distortions are expected (e.g., appropriate sharpness filters or fine-tuning of color saturation) and may have a positive effect on perceptual quality. Conversely, unintended distortions such as motion blur or jitter may have a negative effect on perceptual quality. Video distortion metrics are capable of encoding the quality of a frame based on the distortion contained within the frame.

[0047] It should be noted that process 100 is able to sample frames from video content items at one frame per second and feed the frames as input to the distortion subnet of the deep neural network at the original resolution of the frames (e.g., to avoid rescaling artifacts).

[0048] This is also Figure 2 As shown, frames 230 from multiple extracted frames 210 of video content items can be input into a distortion subnet 232, and the distortion subnet 232 can provide an output 234 including the predicted distortion type and the predicted video distortion index of the frames 230.

[0049] It should be noted that the distortion subnet can be trained on synthetically distorted images from datasets such as KADIS-700K and KADID-10K. Such datasets provide the original image and multiple distortion filters, such as sharpening, denoising, and Gaussian blur, where each filter generates five different levels of distortion (e.g., 125 distortion variants per original). Continuing this example, the training loss function can include a cross-entry loss for multi-label (distortion type) classification. Paired hinge loss between two randomly selected variants with the same distortion type and L2 distance loss The three parts are: cross-entry loss and paired hinge loss. The combination of these factors can be used as the loss for training the initial distortion subnet. The distortion subnet can then be fine-tuned using the KADID-10K dataset. Since the KADID-10K dataset provides ground truth mean opinion scores (MOS), process 100 can use separate multilayer perceptron heads to predict the mean opinion score and use L2 distance loss. Training is then performed. Therefore, the total loss function used to train the distorted subnet on KADID-10K can be expressed as:

[0050]

[0051] Figure 4 The document illustrates illustrative examples of distortion types detected in one or more frames of each video content item, according to some embodiments of the disclosed subject matter. Figure 4As shown, each frame 410, 420, and 430 from different video content items can be presented in the interface along with the predicted distortion type and its associated probability. For example, each frame 410, 420, and 430 can be presented along with the detected distortion type 412 for each frame, such as: distortion type "jitter", distortion type "color quantization", distortion type "lens blur", and distortion type "denoising" for frame 410; distortion type "white noise in color components", distortion type "mean shift", and distortion type "contrast change" for frame 420; and distortion type "denoising", distortion type "JPEG 2000", distortion type "quantization", and distortion type "lens blur" for frame 430.

[0052] Figure 6 The document also illustrates illustrative examples of predicted distortion types detected for frames of video content items, and video distortion metrics for those frames, according to some embodiments of the disclosed subject matter. Figure 6 As shown, an interface is capable of presenting a representative frame 610 including video content items and a predicted distortion type 640 (e.g., "Gaussian blur", "multiplicative noise", and "color saturation") already detected within frame 610. Also as... Figure 6 As shown, the interface can also include a video distortion metric (DT) 650 for frame 610. As described above, the video distortion metric 650, computed by a distortion subnet of a deep neural network, can encode distortion-sensitive embedded features. It should be noted that the video distortion metric 650 can be based on any suitable scale (e.g., a scale from 1 to 5, where 1 is the lowest quality and 5 is the highest quality). For example, the distortion subnet of the deep neural network has determined a video distortion metric of 3.16 for a representative frame 610 of video content items, indicating that frames 610 including “Gaussian blur,” “multiplicative noise,” and “color saturation” are considered to cause a poor viewing experience. In a more specific example, the distortion subnet of the deep neural network can determine the correlation between content and quality—for example, 52 video clips have the content label “strategy video game,” of which 65% are in the high-quality range, while 7 video clips have the content label “forest,” of which 72% are in the low-quality range. In this way, the video distortion index 650 calculated by the distortion subnet of the deep neural network can be based on content tags that describe the content appearing in frame 610.

[0053] Back Figure 1In some embodiments, at point 150, process 100 can feed one or more frames of a video content item as input to a compression subnet of a deep neural network to determine one or more compression levels and compression sensitivity metrics. It should be noted that content-sharing platforms can transcode existing video content items—such as user-generated videos—to different bitrates and / or resolutions to meet device and / or network requirements. For example, compression artifacts can be introduced by third parties—such as upload applications on content-sharing platforms. It should also be noted that video compression strategies often result in noticeable quality degradation. Thus, in some embodiments, the compression subnet can learn compression-related features, wherein the output of the compression subnet can include compression-sensitive embedding features and a successive compression level score or compression sensitivity metric.

[0054] It should be noted that the compression sensitivity index can be any suitable scale. For example, the compression sensitivity index can be a score on the range of 0 to 1, where 0 can represent no compression and 1 can represent heavy compression.

[0055] It should also be noted that process 100 is able to sample multiple frames from the video content item at five frames per second (e.g., to capture both spatial and temporal artifacts) and feed the frames as input to the compressed subnet of the deep neural network at the original resolution of the frames (e.g., to avoid rescaling artifacts).

[0056] This is also Figure 2 As shown, frames 240 from multiple extracted frames 210 of video content items can be input into a compression subnet 242, and the compression subnet 242 can provide an output 244 including a predicted compression level and a predicted compression sensitivity index for frames 240. Figure 2 As shown, it should be noted that the input to each subnet can be different—for example, content subnet 222 can receive entire frames to perform overall semantic classification, distortion subnet 232 can receive single frames, and compression subnet 242 can receive multiple frames to capture both spatial and temporal artifacts. It should also be noted that, as Figure 2 As shown in frames 230 and 240, a frame can be divided into non-overlapping patches for feature extraction, and then the patch features can be stitched together to obtain the features of the entire frame.

[0057] It should be noted that the compressed subnet can be trained using self-supervised learning methods or any other suitable approach. For example, the compressed subnet can be trained by a distilled 3D model pre-trained on a large-scale video classification dataset to learn both spatial and temporal video features. An illustrative example of a large-scale video classification dataset is the Kinetics-600 dataset, which consists of approximately 500,000 10-second video clips already annotated with one of 600 action categories.

[0058] For example, a video collection can be transcoded into multiple variants using the VP9 encoder, such as a Video on Demand (VOD) version, a Video on Demand (VODLB) version with a lower bitrate, and a Constant Bitrate (CBR) version. The VOD version can achieve better quality than the VODLB version due to its higher target bitrate, while the CBR version can achieve the lowest quality because one transcoding step is less optimized than two. Frames can then be uniformly sampled from the initial and transcoded segments and fed into a distilled 3D model to obtain the predicted compression level. An A(1,4,4,100) feature layer, which can be inserted before the fully connected layer, can be used to extract compression-sensitive embedding features.

[0059] Continuing with this example, it should be noted that the training loss function used to train the compressed subnet can include pairwise losses. and comparative loss The two parts. Specifically, the pairwise loss can be calculated to assess the difference in compression level between the original version and the constant bit rate version, and this loss can be expressed as:

[0060] Where K = 4.

[0061] The contrastive loss can be computed by projecting features into a 1D space (1×1600) using two dense layers (to form a non-linear mapping) and defining the similarity (sim(x,y)) between the two features by their feature distance. The contrastive loss can be expressed as:

[0062]

[0063] Therefore, the total loss function used to train the compressed subnet can be expressed as:

[0064]

[0065] Figure 5 The illustration shows illustrative examples of compression levels detected in multiple frames of each video content item according to some embodiments of the disclosed subject matter. Figure 5As shown, representative frames 510, 520, and 530 from different video content items can be presented in the interface along with compression level scores ranging from 0 (indicating no compression) to 1 (indicating heavy or high compression). For example, each representative frame 510, 520, and 530 can be presented along with a predicted compression level score 512, such as a compression level score of 0.892 for representative frame 510, indicating a high compression level. In another example, as an addition to or alternative to the compression level score 512, each representative frame 510, 520, and 530 can be presented along with a textual description of the compression level—such as low, medium, or high compression level.

[0066] Figure 6 The document also illustrates illustrative examples of predicted compression levels and compression sensitivity metrics for input frames of video content items according to some embodiments of the disclosed subject matter. Figure 6 As shown, an interface is available that can present a representative frame 610 including video content items and a portion of the video content items represented by frame 610 with a predicted compression level of 0.924. Also as... Figure 6 As shown, the interface can also include a compression sensitivity index (CP) 670 for the portion of the video content item represented by frame 610. As described above, the compression sensitivity index 670, calculated by the compression subnet of the deep neural network, can encode compression-sensitive embedding features. It should be noted that the compression sensitivity index 670 can be based on any suitable scale (e.g., a scale of 1 to 5, where 1 is the lowest quality and 5 is the highest quality). For example, the compression subnet of the deep neural network has determined that the compression sensitivity index of the representative frame 610 of the video content item is 2.862, indicating that frame 610 with a high compression level of 0.924 is considered to result in a poor viewing experience. That is, although frame 610 is considered to contain interesting content such as “dance” and “outdoor entertainment” with a content quality index of 3.621, process 100 can determine that frame 610 is highly compressed with a compression sensitivity index of 2.862 and contains distortions such as blur and noise with a video distortion index of 3.16, which may lead to a poor viewing experience.

[0067] Back Figure 1In some embodiments, in response to obtaining the content quality metric, video distortion metric, and compression sensitivity metric of a frame of a video content item, process 100 can generate the quality level of the frame of the video content item by concatenating the content quality metric, video distortion metric, and compression sensitivity metric at 160. In some embodiments, process 100 can calculate the quality level of a specific frame of the video content item by averaging the content quality metric, video distortion metric, and compression sensitivity metric. In some embodiments, process 100 can calculate the quality level of a specific frame of the video content item by applying any suitable weight to each of the content quality metric, video distortion metric, and compression sensitivity metric.

[0068] In some embodiments, at 170, process 100 is able to generate an overall quality level for the video content item by aggregating the extracted features to obtain an overall video quality estimate. For example, in some embodiments, process 100 is able to calculate the overall quality level of the video content item by averaging the frame quality scores determined at 160 across the frames of the video content item. In another example, in some embodiments, process 100 is able to use an aggregation subnetwork of a deep neural network to determine the overall quality level. The aggregation subnetwork can be an average pooling (Avg Pool) model, a long short-term model (LSTM), a convolutional long short-term model (ConvLSTM), or any other suitable temporal model. Figure 2 In a more specific example, in the average pooling model of the aggregation subnet 260, a convolutional neural network can be used to generate the overall quality level. This convolutional neural network outputs a block score for each frame out of multiple frames of the video content item and averages the block scores. In the average pooling model, each block feature 250 can be filtered by a 1×1 2D convolutional layer (256 units) to refine the feature space. Those refined features can then be sent through a shared 2D header (which can include batch normalization, rectified linear activation function (ReLU)), global max pooling (GlobalMaxPool2D), dropout, and / or dense layers) to obtain the block score. The average of the block scores can be determined as the overall quality level of the video content item.

[0069] Figure 6 The document also illustrates illustrative examples of the overall quality level of one or more video content items according to some embodiments of the disclosed subject matter. Figure 6As shown, an interface is available that can present a representative frame 610 including video content items and an overall quality level (CP+CT+DT) 680 of 2.955 for the video content items represented by frame 610. It should be noted that a score of 2.955 indicates low perceived quality of the video content items, which may result in a poor viewing experience.

[0070] It should be noted that the quality metrics for one or more frames and the overall video quality estimate can be used in any suitable application.

[0071] For example, process 100 can transmit an overall video quality estimate to the service provider, which allows the service provider to use the overall video quality estimate to make informed quality-of-service trade-offs (e.g., transmitting video content items at relatively high or low resolution or at relatively high or low bitrates). Continuing this example, the service provider can perform adjusted bitrate streaming of video content items transmitted to the user equipment based on the overall video quality estimate for each video content item and / or based on quality metrics for one or more frames of each video content item.

[0072] In another example, process 100 is able to transmit an overall video quality estimate to a service provider, which allows the service provider to determine, using quality metrics associated with each frame of a video content item, that viewers of a first portion of a video content item with a higher quality metric can benefit from being transmitted in a format with a relatively high resolution (e.g., the original content), and that a second portion of a video content item with a lower quality metric can be provided in a format with a relatively low resolution that has little perceptible impact on that portion of the video content item.

[0073] In yet another example, process 100 can transmit an overall video quality estimate to the service provider, which provides an understanding of the quality of video content items, enabling the service provider to optimize the recommendation system (e.g., if multiple videos are presented at a single event) or further compress video content items presented to viewers with low-quality scores that have little perceptible impact on the video content item. Continuing with this example, process 100 can allow the service provider to determine whether the format should be changed from a first format to a potential second format based on the overall video quality estimate—such as the difference in overall video quality estimates between a video content item and subsequent video content items.

[0074] In a further example, process 100 can use extracted features and quality metrics from one or more frames to provide feedback to the uploading user about the video content item, wherein the extracted features (e.g., content labels such as "dance" and "musical ensemble" and distortion types such as "Gaussian blur" and "color saturation") can provide the uploading user with labels of the perceived quality of one or more frames of the video content item. Furthermore, the uploading user can modify the video content item for uploading to the content sharing service (e.g., removing or reducing the types of distortion detected in one or more frames of the video content item to improve the overall perceived quality of the video content item).

[0075] Go to Figure 7 An illustrative example 700 of hardware for determining perceptual quality metrics of video content items, which can be used according to some embodiments of the disclosed subject matter, is shown. As shown, hardware 700 may include a server 702, a communication network 704, and / or one or more user devices 706, such as user devices 708 and 710.

[0076] Server 702 can be any suitable server for storing information, data, programs, media content, and / or any other suitable content. In some embodiments, server 702 can perform any suitable function. For example, in some embodiments, server 702 can determine the overall quality level of the uploaded video content item, the content tags of the content appearing within the frames of the video content item, the type of distortion detected within the frames of the video content item, and the compression level of the frames of the video content item, as described above. Figure 1 Continuing with this example, server 702 can then optimize its recommendation system based on the overall quality level. Server 702 can also determine whether to further compress low-quality original videos that have little or no perceptible impact on the final result by determining whether the format of the video content item will be switched to a different format, in order to select the format of the video content item, and / or determine any other suitable information, such as... Figure 1 The above and the image shown are combined Figure 1 As stated above.

[0077] In some embodiments, communication network 704 can be any suitable combination of one or more wired and / or wireless networks. For example, communication network 704 can include any one or more of the Internet, intranet, wide area network (WAN), local area network (LAN), wireless network, digital subscriber line (DSL) network, frame relay network, asynchronous transfer mode (ATM) network, virtual private network (VPN), and / or any other suitable communication network. User equipment 706 can be connected to communication network 704 via one or more communication links (e.g., communication link 712), and communication network 704 can be linked to server 702 via one or more communication links (e.g., communication link 714). Communication links can be any communication link suitable for transmitting data between user equipment 706 and server 702, such as network links, dial-up links, wireless links, hardwired links, any other suitable communication links, or any suitable combination of such links.

[0078] User equipment 706 may include any one or more user devices suitable for uploading and / or streaming media content from server 702. In some embodiments, user equipment 706 may include any suitable type of user device, such as mobile phones, tablets, wearable computers, laptops, desktop computers, smart TVs, media players, game consoles, vehicle information and / or entertainment systems, and / or any other suitable type of user device. In some embodiments, user equipment 706 may initiate the uploading of video content items to server 702 or any other suitable content sharing server, and in response, user equipment 706 may receive the overall quality level of the uploaded video content item, as well as the content tags of the content appearing within the frames of the video content item, the type of distortion detected within the frames of the video content item, and the compression level of the frames of the video content item, as described above. Figure 1 The user equipment 706 can then determine whether to modify the uploaded video content items (e.g., to improve the overall quality level by modifying the uploaded video content items to reduce or remove distortion).

[0079] Although server 702 is shown as a single device, in some embodiments, any suitable number of devices can be used to perform the functions performed by server 702. For example, in some embodiments, multiple devices can be used to implement the functions performed by server 702.

[0080] although Figure 7 Two user devices 708 and 710 are shown to avoid making the diagram too complex, but in some embodiments any suitable number of user devices and / or any suitable type of user device can be used.

[0081] In some embodiments, server 702 and user equipment 706 can be implemented using any suitable hardware. For example, in some embodiments, devices 702 and 706 can be implemented using any suitable general-purpose computer or special-purpose computer. For example, a mobile phone can be implemented using a special-purpose computer. Any such general-purpose computer or special-purpose computer can include any suitable hardware. For example, such as Figure 8 As shown in example hardware 800, such hardware may include a hardware processor 802, a memory and / or storage device 804, an input device controller 806, an input device 808, a display / audio driver 810, a display and audio output circuit 812, a communication interface 814, an antenna 816, and a bus 818.

[0082] In some embodiments, hardware processor 802 may include any suitable hardware processor, such as a microprocessor, microcontroller, digital signal processor, dedicated logic, and / or any other suitable circuitry for controlling the operation of a general-purpose computer or a special-purpose computer. In some embodiments, hardware processor 802 may be controlled by a server program stored in the memory and / or storage of a server, such as server 702. In some embodiments, hardware processor 802 may be controlled by a computer program stored in the memory and / or storage 804 of user equipment 706.

[0083] In some embodiments, memory and / or storage device 804 can be any suitable memory and / or storage device for storing programs, data, and / or any other suitable information. For example, memory and / or storage device 804 can include random access memory, read-only memory, flash memory, hard disk storage, optical media, and / or any other suitable memory.

[0084] In some embodiments, the input device controller 806 can be any suitable circuitry for controlling and receiving input from one or more input devices 808. For example, the input device controller 806 can be circuitry for receiving input from a touchscreen, a keyboard, one or more buttons, a voice recognition circuit, a microphone, a camera, an optical sensor, an accelerometer, a temperature sensor, a near-field sensor, a pressure sensor, an encoder, and / or any other type of input device.

[0085] In some embodiments, the display / audio driver 810 can be any suitable circuitry used to control and drive the output to one or more display / audio output devices 812. For example, the display / audio driver 810 can be circuitry used to drive a touchscreen, flat panel display, cathode ray tube display, projector, one or more speakers, and / or any other suitable display and / or presentation device.

[0086] The communication interface 814 can be any suitable circuit for interface connection to one or more communication networks (e.g., computer network 704). For example, interface 814 can include network interface card circuitry, wireless communication circuitry, and / or any other suitable type of communication network circuitry.

[0087] In some embodiments, antenna 816 can be any suitable one or more antennas for wireless communication with a communication network (e.g., communication network 704). In some embodiments, antenna 416 can be omitted.

[0088] In some embodiments, bus 818 can be any suitable mechanism for communication between two or more components 802, 804, 806, 810 and 814.

[0089] According to some embodiments, any other suitable component can be included in the hardware 800.

[0090] In some embodiments, at least some of the blocks of the processes in the above figures can be implemented or executed in any order or sequence, which is not limited to the order and sequence shown in and described in conjunction with the figures. Furthermore, it is possible to implement or execute them substantially simultaneously or in parallel where appropriate. Figure 1 Some of the points in the boxes above are used to reduce latency and processing time. Additionally or alternatively, the above can be omitted. Figure 1 Some of the process frames.

[0091] In some embodiments, any suitable computer-readable medium can be used to store instructions for performing the functions and / or processes described herein. For example, in some embodiments, the computer-readable medium can be transient or non-transient. For example, a non-transient computer-readable medium can include media such as non-transient magnetic media (such as hard disks, floppy disks, and / or any other suitable magnetic media), non-transient optical media (such as compact disks, digital video disks, Blu-ray discs, and / or any other suitable optical media), non-transient semiconductor media (such as flash memory, electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and / or any other suitable semiconductor media), any suitable medium that does not vanish during transmission or has no permanent appearance, and / or any suitable tangible medium. As another example, a transient computer-readable medium can include signals in networks, in wires, conductors, optical fibers, circuits, any suitable medium that vanishes during transmission and has no permanent appearance, and / or any suitable intangible medium.

[0092] In situations where the systems discussed herein collect or may utilize personal information about users, users may be given the opportunity to control whether procedures or features collect user information (e.g., information about the user's social networks, social actions or activities, occupation, user preferences, or the user's current location) and / or to control whether and / or how content that may be more relevant to the user is received from the content server. Additionally, certain data may be processed in one or more ways before it is stored or used, resulting in the removal of personally identifiable information. For example, a user's identity may be processed to the point that the user's personally identifiable information cannot be determined, or, where location information is available, the user's geographic location may be generalized (e.g., to the city, zip code, or state level), making the user's specific location impossible to determine. Therefore, users may have control over how information about themselves is collected and used by the content server.

[0093] Therefore, methods, systems, and media for determining the perceived quality metrics of video content items are provided.

[0094] Although the invention has been described and illustrated in the foregoing illustrative embodiments, it should be understood that this disclosure is made by way of example only, and many changes in the details of the embodiments of the invention can be made without departing from the spirit and scope of the invention, which is defined only by the appended claims. Features of the disclosed embodiments can be combined and rearranged in various ways.

Claims

1. A method for video quality assessment, the method comprising: Receive video content items; Extract multiple frames from the video content item; A first subnet of a deep neural network is used to determine a content quality metric for each of the plurality of frames of the video content item, wherein the content quality metric represents the semantic level embedding of each of the plurality of frames of the video content item. The second subnet of the deep neural network is used to determine the video distortion index for each of the plurality of frames of the video content item; The third subnet of the deep neural network is used to determine the compression sensitivity index of each of the plurality of frames of the video content item. The quality level of each frame in the plurality of frames of the video content item is generated, the quality level linking the content quality index, the video distortion index and the compression sensitivity index of the frame of the video content item; A convolutional neural network is used to generate an overall quality level for the video content item by aggregating the quality levels of each frame from the plurality of frames. The convolutional neural network outputs a block score for each frame from the plurality of frames of the video content item and averages the block scores. This allows video recommendations to be presented based on the overall quality level of the video content items.

2. The method according to claim 1, wherein, The first subnet of the deep neural network further outputs predicted content labels, which describe the content appearing in each of the plurality of frames of the video content item.

3. The method according to claim 1, wherein, The video distortion metric represents the distortion-sensitive embedding of each of the plurality of frames of the video content item, and wherein the second subnetwork of the deep neural network further outputs a detected distortion type, the detected distortion type describing the distortion detected in each of the plurality of frames of the video content item.

4. The method according to claim 1, wherein, The compression sensitivity metric represents the compression-sensitive embedding of each of the plurality of frames of the video content item, and wherein the third subnet of the deep neural network further outputs a compression level score.

5. The method according to claim 1, wherein, The video recommendation includes recommendations for further compressing the video content items based on the overall quality level.

6. The method according to claim 1, wherein, The video recommendations include recommendations for the uploader of the video content item to modify a portion of the video content item based on the quality level associated with the frame of the video content item.

7. A system for video quality assessment, the system comprising: Hardware processor, the hardware processor being configured to: Receive video content items; Extract multiple frames from the video content item; A first subnet of a deep neural network is used to determine a content quality metric for each of the plurality of frames of the video content item, wherein the content quality metric represents the semantic level embedding of each of the plurality of frames of the video content item. The second subnet of the deep neural network is used to determine the video distortion index for each of the plurality of frames of the video content item; The third subnet of the deep neural network is used to determine the compression sensitivity index of each of the plurality of frames of the video content item. The quality level of each frame in the plurality of frames of the video content item is generated, the quality level linking the content quality index, the video distortion index and the compression sensitivity index of the frame of the video content item; A convolutional neural network is used to generate an overall quality level for a video content item by aggregating the quality levels of each frame from the plurality of frames. The convolutional neural network outputs a block score for each frame from the plurality of frames of the video content item and averages the block scores. This allows video recommendations to be presented based on the overall quality level of the video content items.

8. The system according to claim 7, wherein, The first subnet of the deep neural network further outputs predicted content labels, which describe the content appearing in each of the plurality of frames of the video content item.

9. The system according to claim 7, wherein, The video distortion metric represents the distortion-sensitive embedding of each of the plurality of frames of the video content item, and wherein the second subnetwork of the deep neural network further outputs a detected distortion type, the detected distortion type describing the distortion detected in each of the plurality of frames of the video content item.

10. The system according to claim 7, wherein, The compression sensitivity metric represents the compression-sensitive embedding of each of the plurality of frames of the video content item, and wherein the third subnet of the deep neural network further outputs a compression level score.

11. The system according to claim 7, wherein, The video recommendation includes recommendations for further compressing the video content items based on the overall quality level.

12. The system according to claim 7, wherein, The video recommendations include recommendations for the uploader of the video content item to modify a portion of the video content item based on the quality level associated with the frame of the video content item.

13. A non-transitory computer-readable medium comprising computer-executable instructions, which, when executed by a processor, cause the processor to perform a method for video quality assessment, the method comprising: Receive video content items; Extract multiple frames from the video content item; A first subnet of a deep neural network is used to determine a content quality metric for each of the plurality of frames of the video content item, wherein the content quality metric represents the semantic level embedding of each of the plurality of frames of the video content item. The second subnet of the deep neural network is used to determine the video distortion index for each of the plurality of frames of the video content item; The third subnet of the deep neural network is used to determine the compression sensitivity index of each of the plurality of frames of the video content item. The quality level of each frame in the plurality of frames of the video content item is generated, the quality level linking the content quality index, the video distortion index and the compression sensitivity index of the frame of the video content item; A convolutional neural network is used to generate an overall quality level for a video content item by aggregating the quality levels of each frame from the plurality of frames. The convolutional neural network outputs a block score for each frame from the plurality of frames of the video content item and averages the block scores. This allows video recommendations to be presented based on the overall quality level of the video content items.

14. The non-transitory computer-readable medium according to claim 13, wherein, The first subnet of the deep neural network further outputs predicted content labels, which describe the content appearing in each of the plurality of frames of the video content item.

15. The non-transitory computer-readable medium according to claim 13, wherein, The video distortion metric represents the distortion-sensitive embedding of each of the plurality of frames of the video content item, and wherein the second subnetwork of the deep neural network further outputs a detected distortion type, the detected distortion type describing the distortion detected in each of the plurality of frames of the video content item.

16. The non-transitory computer-readable medium according to claim 13, wherein, The compression sensitivity metric represents the compression-sensitive embedding of each of the plurality of frames of the video content item, and wherein the third subnet of the deep neural network further outputs a compression level score.

17. The non-transitory computer-readable medium according to claim 13, wherein, The video recommendation includes recommendations for further compressing the video content items based on the overall quality level.

18. The non-transitory computer-readable medium according to claim 13, wherein, The video recommendations include recommendations for the uploader of the video content item to modify a portion of the video content item based on the quality level associated with the frame of the video content item.

Images