A method, device and medium for format selection

Abstract

A system and method for selecting an optimal format for a video player based on visual quality is disclosed. The method includes: generating multiple reference transcoded versions of a reference video; obtaining quality scores of frames of the multiple reference transcoded versions of the reference video; generating a first training input including a set of color attributes, spatial attributes, and temporal attributes of frames of the reference video; and generating a first target output for the first training input, wherein the first target output includes the quality scores of frames of the multiple reference transcoded versions of the reference video. The method further includes: providing training data to train a machine learning model on (i) a training input set including the first training input and (ii) a target output set including the first target output.

Description

Technical Field

[0001] Aspects and embodiments of the present disclosure relate to video processing, and more particularly, to optimal format selection for a video player based on predicted visual quality. Background Art

[0002] Content sharing platforms enable users to upload, consume, search, approve ("like"), dislike, and / or comment on content such as videos, images, audio clips, news stories, and the like. In a content sharing platform, a user can upload content (e.g., videos, images, audio clips, and the like) to be included in the platform, thereby enabling other users to consume (e.g., view, etc.) the content. Most content sharing platforms transcode the original source video from its original encoding format to a common format. Transcoding includes: decoding the source video from the original format to an unencoded representation using a codec in the original format; and then encoding the unencoded representation using a codec in the common format. Transcoding can be used to reduce storage requirements, and can also be used to reduce bandwidth requirements for providing video to clients. Summary of the invention

[0003] A simplified summary of various aspects of the present disclosure is presented below to provide a basic understanding of these aspects. This summary is not an extensive overview of all anticipated aspects and is neither intended to identify important or critical elements nor to describe the scope of these aspects. Its purpose is to present some concepts of the present disclosure in a simplified form as a prelude to a more detailed description presented later.

[0004] In one aspect of the present disclosure, a system and multiple methods are disclosed for training a machine learning model (e.g., a neural network, a convolutional neural network (CNN), a support vector machine [SVM], etc.) and processing a video using the trained model. In one embodiment, a method includes: generating training data for a machine learning model to be trained to identify quality scores of a set of transcoded versions of a new video at a display resolution set. Generating the training data may include: generating multiple reference transcoded versions of a reference video; obtaining quality scores of frames of the multiple reference transcoded versions of the reference video; generating a first training input including a set of color attributes, spatial attributes, and temporal attributes of frames of the reference video; and generating a first target output for the first training input, wherein the first target output includes quality scores of frames of the multiple reference transcoded versions of the reference video. The method further includes: providing training data to train the machine learning model on (i) a training input set including the first training input and (ii) a target output set including the first target output.

[0005] In one embodiment, the quality score comprises a peak signal to noise ratio (PSNR) of the frame. In some embodiments, the quality score comprises a video multi-method assessment fusion (VMAF) of the frame. In addition, the color attribute may include at least one of the RGB or Y values ​​of the frame. The spatial attribute may include a Gabor feature filter bank. The temporal attribute may include optical flow.

[0006] In some embodiments, the machine learning model is configured to process the new video and generate one or more outputs indicating quality scores of a set of transcoded versions of the new video at a set of display resolutions. In addition, the multiple transcoded versions of the reference video may include transcodes of the reference video at each of a plurality of different video resolutions, transcoding configurations, and different display resolutions.

[0007] In addition, a computing device for performing the operations of the above methods and various embodiments described herein is disclosed. A computer-readable medium storing instructions for performing the operations associated with the above methods and various embodiments described herein is also disclosed. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The various aspects and embodiments of the present disclosure will be more fully understood from the detailed description given below and from the drawings of the various aspects and embodiments of the present disclosure, which, however, should not be considered to limit the present disclosure to certain aspects or embodiments but are only for explanation and understanding.

[0009] Figure 1 An illustrative computer system architecture is depicted in accordance with one or more aspects of the present disclosure.

[0010] Figure 2 is a block diagram of an example training set generator for creating a dataset for a machine learning model using a historical video dataset according to one or more aspects of the present disclosure.

[0011] Figure 3 is a block diagram illustrating a system for determining a video classification in accordance with one or more aspects of the present disclosure.

[0012] Figure 4 A flowchart of one example of a method for training a machine learning model according to one or more aspects of the present disclosure is depicted.

[0013] Figure 5 A flow chart of one example of a method for processing a video using a trained machine learning model according to one or more aspects of the present disclosure is depicted.

[0014] Figure 6A flowchart of one example of a method for processing a video using a trained machine learning model and using the output of the trained machine learning model to optimize format selection at a server device in accordance with one or more aspects of the present disclosure is depicted.

[0015] Figure 7 A flow chart of one example of a method for optimizing format selection using the output of a trained machine learning model at a client device in accordance with one or more aspects of the present disclosure is depicted.

[0016] Fig. 8A and 8B An example graphical representation of the output of a trained machine learning model for optimizing format selection according to an embodiment of the present disclosure is provided.

[0017] Fig. 9 A block diagram of an illustrative computer system that operates in accordance with one or more aspects of the present disclosure is depicted. DETAILED DESCRIPTION

[0018] In a content sharing platform, users can upload content (e.g., videos, images, audio clips, etc.) to be included in the platform, thereby enabling other users to consume (e.g., view, etc.) the content. Due to limitations of viewing devices and network bandwidth, videos uploaded to the content sharing platform are transcoded (decompressed and recompressed) before being provided to viewers in order to enhance the viewing experience. Uploaded videos may have multiple transcoded variants that play at various display resolutions. Resolutions (input, transcoding, and display) can be roughly grouped into standardized industry standard resolutions, such as 360p, 480p, 720p, 1080p, and 2160p (4k), etc.

[0019] A typical transcoding pipeline is capable of generating multiple transcoded versions (also called video formats). When playing a video, the media viewer is able to adaptively select one of those video formats to serve. Assuming the user has enough bandwidth, the traditional service strategy is to switch to a higher resolution version of the video until the highest available resolution is reached. This is also known as an adaptive bitrate (ABR) strategy. The assumption of this ABR strategy is that the higher resolution version provides better visual quality. However, in some cases, the visual quality of the higher resolution version can be very close to the visual quality of the lower resolution version (for example, when a 480p version of the video has similar perceptual quality to a 720p version of the video when played on a client device with a 480p display resolution). In this case, providing a higher resolution version of the video wastes the user's bandwidth without providing the user with a significant benefit in terms of perceptual quality.

[0020] One means of avoiding this inefficiency due to suboptimal format selection is to append a list of objective quality scores to each transcoded version, and implicitly assume that each objective quality score reflects the perceived quality when a particular format is played at a particular display resolution. Computing a single quality score for a particular video format requires decoding both video streams (i.e., the rescaled transcoded version and the original version), and extracting per-frame features in order to compute an overall quality score. If it is assumed that there are 'N_format' valid transcoded video formats, 'N_transconfig' candidate transcoding configurations for transcoding the input video to a certain video format, and 'N_display' possible display resolutions, then the total number of possible quality scores to be calculated will be: 'N_format' x 'N_transconfig' x 'N_display' scores. Computing all possible quality scores across various resolutions uses a large amount of computing resources. Furthermore, it may be infeasible to perform such large-scale calculations on large systems (e.g., content sharing platforms with millions of new uploads per day).

[0021] Disclosed herein are aspects and embodiments of systems and methods for effectively predicting all possible video quality scores for a video based on deep learning. More specifically, embodiments relate to training and using effective machine learning models to predict objective quality scores for videos compressed with arbitrary compression settings and played at arbitrary display resolutions.

[0022] According to one embodiment, a historical video set is accessed and used to train a machine learning model. In particular, each of the historical videos is used to generate input features to train the machine learning model. The input features include a set of attributes of a frame of the corresponding historical video. The attribute set can include color attributes (e.g., red / green / blue (RGB) intensity values ​​or Y intensity values ​​in YUV format), spatial attributes (e.g., Gabor filters), and temporal attributes (e.g., optical flow) of (one or more) frames of the corresponding historical video. In addition, each of the historical videos is transcoded into a plurality of different resolution formats and transcoding configurations to generate a plurality of transcoded versions of the video. The plurality of transcoded versions are then rescaled to a plurality of different potential display resolutions of the client device (referred to herein as rescaled transcoded versions). A quality score (e.g., peak signal-to-noise ratio (PSNR) measurement or video multi-method assessment fusion (VMAF measurement)) is obtained for each of the rescaled transcoded versions.

[0023] These quality scores can then be used as training outputs (e.g., ground truth labels), which are mapped to the training input features discussed above and used to train the machine learning model. In this way, the machine learning model is trained to generate predicted quality scores for the video at each possible tuple of video resolution format, transcoding configuration, and display resolution. As used herein, the terms "video resolution format" and "video format" may refer to the resolution of the video before rescaling. The term "display resolution" may refer to the resolution at which the video is actually displayed after rescaling (e.g., by a media viewer on a client device).

[0024] After the machine learning model has been trained, new videos can be identified for processing by the trained machine learning model. In this case, the perceptual quality of the constituent videos (e.g., the videos provided to the client device for playback) at various video resolutions and the transcoding configurations at various display resolutions are unknown because the new video is provided to the machine learning model in its entirety without knowing what the perceptual quality of the video will be on a particular client device.

[0025] In one embodiment, a set of attributes (e.g., color, spatial, temporal) of a new video is determined. The set of attributes of the new video is presented as input to a trained machine learning model, which generates one or more outputs based on the input. In one embodiment, the output is a predicted quality score that provides a predicted perceptual quality measure of the video at each possible tuple of video resolution, transcoding configuration, and display resolution. In some embodiments, the predicted quality score can be used to optimize format selection at a client device. Certain aspects of the training and use of the machine learning model are described in more detail below.

[0026] Aspects of the present disclosure therefore provide a mechanism by which predicted quality scores for videos at all possible combinations of video resolution, transcoding configuration, and display resolution can be identified. The mechanism allows automatic and optimized format selection for video playback at a client device with a specific display resolution. An advantage of an embodiment of the present disclosure is that a trained machine learning model can return multiple objective video quality scores (e.g., PSNR and VMAF values) for all (video_format, transcoding_config, display_resolution) tuples of an input video at once. The embodiment avoids the process of time-consuming transcoding or quality metric calculation for each possible transcoded version of the input video. The output of the trained machine learning model can be used to optimize format selection to maximize the user experience of video quality. Optimizing format selection can also have the advantage of reducing bandwidth requirements without significantly reducing the video quality perceived by the user.

[0027] Figure 1 An illustrative system architecture 100 is shown according to one embodiment of the present disclosure. The system architecture 100 includes one or more server machines 120 to 150, a content repository 110, and client machines 102A-102N connected to a network 104. The network 104 can be a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), or a combination thereof.

[0028] The client machines 102A-102N may be personal computers (PCs), laptops, mobile phones, tablet computers, set-top boxes, televisions, video game consoles, digital assistants, or any other computing devices. The client machines 102A-102N may run an operating system (OS) that manages the hardware and software of the client machines 102A-102N. In one embodiment, the client machines 102A-102N may upload videos to a web server (e.g., upload server 125) for storage and / or processing.

[0029] The server machines 120 to 150 may be rack-mounted servers, router computers, personal computers, portable digital assistants, mobile phones, laptop computers, tablet computers, cameras, camcorders, netbooks, desktop computers, media centers, or any combination thereof. The server machines 120 include an upload server 125 capable of receiving content (e.g., videos, audio clips, images, etc.) uploaded by client machines 102A-102N (e.g., via a web page, via an application, etc.).

[0030] The content repository 110 is a persistent storage capable of storing content items and data structures for tagging, organizing, and indexing media items. The content repository 110 may be hosted by one or more storage devices, such as main memory, magnetic or optical storage-based disks, tapes, or hard drives, NAS and SAN, etc. In some implementations, the content repository 110 may be a network-attached file server, while in other embodiments, the content repository 110 may be some other type of persistent storage, such as an object-oriented database and a relational database, etc., which may be hosted by the server machine 120 or one or more different machines coupled to the server machine 120 via the network 104.

[0031] Content items stored in content repository 110 may include user-generated media items uploaded by client machines, as well as media items from service providers such as news organizations, publishers, and libraries. In some implementations, content repository 110 may be provided by a third-party service, while in some other implementations, content repository 110 may be maintained by the same entity that maintains server machine 120. In some examples, content repository 110 and server machines 120-150 may be part of a content sharing platform that allows users to upload, consume, search, approve ("like"), dislike, and / or comment on media items.

[0032] The content sharing platform may include multiple channels. A channel can be data content available from a public source or data content with a public topic, theme or theme. The data content can be digital content selected by a user, digital content available to a user, digital content uploaded by a user, digital content selected by a content provider, digital content selected by a broadcaster, etc. A channel can be associated with an owner, who is a user who can perform activities on a channel. Different activities can be associated with a channel based on the owner's activities, such as the owner providing digital content on a channel, the owner selecting (e.g., liking) digital content associated with another channel, the owner commenting on digital content associated with another channel, etc. Activities associated with a channel can be collected into an activity feed of a channel. Users other than channel owners can subscribe to one or more channels they are interested in. The concept of "subscription" may also be referred to as "like", "follow", and "add friends", etc.

[0033] Each channel may include one or more media items. Examples of media items can include, but are not limited to, digital videos, digital movies, digital photos, digital music, website content, social media updates, e-books, e-magazines, digital newspapers, digital audiobooks, e-journals, web blogs, Really Simple Syndication (RSS) feeds, electronic comic books, software applications, etc. In some embodiments, media items are also referred to as video content items.

[0034] The media items may be consumed via a media viewer 105 executing on a client machine 102A-102N. In one embodiment, the media viewer 105 may be an application that allows a user to view content such as images, videos (e.g., video content items), web pages, documents, and the like. For example, the media viewer 105 may be a web browser that is capable of accessing, retrieving, presenting, and / or navigating content provided by a network server (e.g., web pages such as Hypertext Markup Language (HTML) pages, digital media items, or content items, etc.). The media viewer 105 may present, display, and / or provide content (e.g., web pages, media viewers) to the user. The media viewer 105 may also display an embedded media player (e.g., a web page that may provide information about products sold by an online merchant) embedded in a web page (e.g., a web page that may provide information about products sold by an online merchant). In another example, the media viewer 105 may be a standalone application (eg, a mobile application) that allows a user to view digital media content items (eg, digital videos, digital images, e-books, etc.).

[0035] The media viewer 105 may be provided to the client devices 102A to 102N by the server 120 and / or the content sharing platform. For example, the media viewer 105 may be an embedded media player embedded in a web page provided by the content sharing platform. In another example, the media viewer may be an application that communicates with the server 120 and / or the content sharing platform.

[0036] Embodiments of the present disclosure provide for training and using effective machine learning models to predict the objective quality of videos compressed at arbitrary compression settings and played at arbitrary display resolutions. The server machine 130 includes a training set generator 131 capable of generating training data (e.g., training input and target output sets) to train such machine learning models. Figure 2 Some operations of the training set generator 131 are described in detail.

[0037] The server machine 140 includes a training engine 141 capable of training a machine learning model 160. The machine learning model 160 may refer to a model artifact created by the training engine 141 using training data including training inputs and corresponding target outputs (correct answers to the corresponding training inputs). The training engine 141 may find patterns in the training data that map the training inputs to the target outputs (answers to be predicted) and provide a machine learning model 160 that captures these patterns. The machine learning model may consist of, for example, a single-stage linear or nonlinear operation (e.g., a support vector machine [SVM]), or may be a deep network, i.e., a machine learning model consisting of multiple stages of nonlinear operations). An example of a deep network is a neural network having one or more hidden layers, and such a machine learning model may be trained by, for example, adjusting the weights of the neural network according to a back-propagation learning algorithm, etc. For convenience, the remainder of the present disclosure refers to the embodiments as neural networks, although some embodiments may employ SVMs or other types of learning machines instead of or in addition to neural networks. In one embodiment, a convolutional neural network (CNN), such as ResNet or EfficientNet, is used as the primary training model for the machine learning model. Other machine learning models may be considered in embodiments of the present disclosure. In one aspect, the training set is obtained from a server machine 130 .

[0038] The server machine 150 includes a quality score engine 151 and a format analysis engine 152. The quality score engine 151 is capable of providing attribute data of a video frame as input to a trained machine learning model 160 and running the trained machine learning model 160 on the input to obtain one or more outputs. Figure 4 As described in detail, in one embodiment, the format analysis engine 152 can also extract quality score data from the output of the trained machine learning model 160 and use the quality score data to perform optimal format selection for the video. In some embodiments, the format analysis engine 152 can be provided by the media viewer 105 at the client device 102A-102N based on the quality score data obtained by the quality score engine 151.

[0039] It should be noted that in some other implementations, the functionality of server machines 120, 130, 140, and 150 may be provided by fewer machines. For example, in some implementations, server machines 130 and 140 may be integrated into a single machine, while in other implementations, server machines 130, 140, and 150 may be integrated into a single machine. Additionally, in some implementations, one or more of server machines 120, 130, 140, and 150 may be integrated into a content sharing platform.

[0040] In general, if appropriate, the functions described as being performed by the content item sharing platform, server machine 120, server machine 130, server machine 140 and / or server machine 150 in one embodiment can also be performed on client devices 102A to 102N in other embodiments. In addition, the functions attributed to a particular component can be performed by different or multiple components operating together. The content sharing platform, server machine 120, server machine 130, server machine 140 and / or server machine 150 can also be accessed as a service provided to other systems or devices through appropriate application programming interfaces, and are therefore not limited to use on websites.

[0041] Figure 2 According to some embodiments, the historical video dataset 240 is used as a basis for the machine learning model (e.g. Figure 1 An example training set generator 231 (e.g., Figure 1 Training set generator 131). Figure 2 The system 200 shows a training set generator 231 , training input features 210 , and target output labels 220 .

[0042] In some embodiments, the training set generator 231 generates a data set (e.g., a training set, a validation set, a test set) that includes one or more training input features 210 (e.g., training input, validation input, test input) and one or more target output labels 220 corresponding to the training input features 210. The data set may also include mapping data that maps the training input features 210 to the target output labels 220. The training input features 210 may also be referred to as “data inputs,” “features,” “attributes,” or “information.” In some embodiments, the training set generator 131 may provide the data set to a training set. Figure 1 A training engine 141, wherein the data set is used to train, validate or test the machine learning model 160. Figure 4 Some embodiments of generating a training set are further described.

[0043] In some embodiments, the training input features 210 may include one or more of historical video color attributes 242, historical video spatial attributes 244, historical video temporal attributes 246, etc. The target output labels 220 may include a video classification 248. The video classification 248 may include or be associated with a video quality score measurement algorithm.

[0044] In some embodiments, the training set generator 231 can generate data inputs corresponding to a feature set (e.g., one or more historical video color attributes 242, historical video spatial attributes 244, historical video temporal attributes 246) to train, validate, or test a machine learning model for each input resolution of a set of possible input resolutions for a video. Thus, each typical resolution of a video (e.g., 360p, 480p, 720p, 1080p, etc.) can have its own model, and any non-standard arbitrary input resolution (e.g., 1922x1084) can be rescaled to the closest standard resolution with the same aspect ratio, with any missing parts filled with 0. For example, a first machine learning model can be trained, validated, and tested for an input video with a resolution of 360p. A second machine learning model can be trained, validated, and tested for an input video with a resolution of 480p. A third machine learning model can be trained, validated, and tested for an input video with a resolution of 720p, and so on.

[0045] The training set generator 231 can utilize the historical video 240 set to train, validate, and test (one or more) machine learning models. In some embodiments, an existing data set of a content sharing platform can be curated and used as a historical video data set 240, specifically for the purpose of training, validating, and testing machine learning models. In one embodiment, the historical video data set 240 may include multiple (e.g., in the order of thousands) videos of short duration (e.g., 20 seconds, etc.) with various input resolutions (e.g., 360p, 480p, 720p, 1080p, and 2160p (4K), etc.). In some embodiments, the data (e.g., videos) in the historical video data set 240 can be divided into training data and test data. For example, the videos of the historical video data set 240 can be randomly divided into 80% and 20% for training and testing, respectively.

[0046] To generate a set of training input features 210 and training output labels 220, the training set generator 231 may iterate the following process for each video in the historical video dataset 240. For ease of reference, the process is described with respect to the first reference video of the historical video dataset 240. It should be understood that the training set generator 231 may perform a similar process for all videos in the historical video dataset 240.

[0047] For the first reference video, the training set generator 231 obtains the training input features 210 of the first reference video. In one embodiment, for each frame of the first reference video, a set of video color attributes 242, video spatial attributes 244, and video temporal attributes 246 are extracted for each frame of the first reference video. The video color attribute 242 may refer to at least one of an RGB intensity value or a Y intensity value (of a Y'UV model) of a pixel of the frame.

[0048] The video spatial attribute 244 may refer to a Gabor filter feature set. A Gabor filter is a linear filter used for texture analysis, which analyzes whether there is any specific frequency content in a frame in a specific direction in a local area around an analysis point or analysis area. In the spatial domain, a 2D Gabor filter is a Gaussian kernel function modulated by a sinusoidal plane wave. Embodiments of the present disclosure may also utilize other spatial features of a frame, such as block-based features (e.g., SSM, VMS, etc.).

[0049] Video temporal attribute 246 may refer to optical flow. Optical flow refers to the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by relative motion between an observer and the scene. Optical flow may be defined as the distribution of apparent motion speeds of brightness patterns in an image. Embodiments of the present disclosure may also utilize other temporal features of frames, such as calculating differences between pixels of adjacent frames.

[0050] The extracted sets of historical video color attributes 242, historical video spatial attributes 244, and historical video temporal attributes 246 for each frame in the first reference video are then combined as a first training input feature set for the first reference video. Other feature sets from frames of the input video may also be considered in embodiments of the present disclosure.

[0051] In order to obtain the target output label 220 of the first reference video, the training set generator 231 can transcode the first reference video to multiple valid video formats (e.g., 360p, 480p, 720p, 1080p, and 2160p (4K), etc.) with multiple transcoding configurations (e.g., using an H.264 encoder to scan the constant rate factor (CRF) from 0 to 51). Other codecs and encoders can be used in other embodiments of the present disclosure, such as the VP9 codec. The training set generator 231 then rescales all transcoded versions to multiple display resolutions (e.g., 360p, 480p, 720p, 1080p, and 2160p (4K), etc.), as well as rescales the original version of the input. Each rescaled transcoded version and the original version are provided to the quality analyzer as a source and a reference to obtain a quality score for each frame of each rescaled transcoded version and the original version of the first reference video.

[0052] In one embodiment, the quality score may be a PSNR measurement of each frame of each rescaled transcoded version and the original version of the first reference video. PSNR refers to the ratio between the maximum possible power of a signal and the power of corrupting noise that affects the fidelity of its representation. PSNR may be expressed in logarithmic decibel scale. In one embodiment, the quality score may be a VMAF measurement of each frame of each rescaled transcoded version and the original version of the first reference video. VMAF refers to a full-reference video quality metric that predicts subject video quality based on reference and distorted video sequences.

[0053] The calculated quality scores are used as video classifications (eg, input ground truth labels) 248 for target output labels 220 for each possible video format, transcoding configuration, and display resolution of the first reference video.

[0054] Figure 3 3 is a block diagram illustrating a system 300 for determining a video classification 346 according to some embodiments. At block 310, the system 300 performs a historical video 342 (e.g., Figure 2 The data of the historical video data set 240) is divided (for example, by Figure 1 The training set generator 131 of the server machine 130 and / or Figure 2 The training set generator 231 of FIG. 20 is used to generate the training set 302, the validation set 304, and the test set 306. For example, the training set may be 60% of the historical video 342, the validation set may be 20% of the historical video 342, and the validation set may be 20% of the historical video 342. Figure 2 As described above, the system 300 can generate multiple feature sets (eg, color attributes, spatial attributes, temporal attributes) for each of the training set, the validation set, and the test set.

[0055] At block 312, system 300 performs model training using training set 302 (e.g., by Figure 1 The system 300 may train multiple models using multiple feature sets of the training set 302 (e.g., a first feature set of the training set 302, a second feature set of the training set 302, etc.). For example, the system 300 may train a machine learning model using a first feature set in the training set (e.g., for a first input resolution, such as 360p) to generate a first trained machine learning model, and may generate a second trained machine learning model using a second feature set in the training set (e.g., for a second input resolution, such as 480p). In some embodiments, many models may be generated, including models with various feature arrangements and combinations of models.

[0056] In some embodiments, model training is performed using a convolutional neural network (CNN) training model. Some examples of CNN training models include ResNet and EfficientNet. In some embodiments, other training models may also be used.

[0057] At block 314, the system 300 performs model validation using the validation set 304. The system 300 can validate each of the trained models using the corresponding feature set of the validation set 304. For example, the system 300 can validate a first trained machine learning model using a first feature set in the validation set (e.g., for a 360p input resolution) and train a second trained machine learning model using a second feature set in the validation set (e.g., for a 480p input resolution). In some embodiments, the system 300 can validate many models generated at block 312 (e.g., models with various arrangements of features, combinations of models, etc.).

[0058] At block 314, the system 300 may determine the accuracy of each of the one or more trained models (e.g., via model validation), and may determine whether the one or more trained models have an accuracy that satisfies a threshold accuracy. For example, a loss function defined based on the absolute difference between the predicted quality and the ground truth may be utilized. Higher-order ||L|| difference specifications may also be considered in other embodiments. In response to determining that none of the trained models have an accuracy that satisfies the threshold accuracy, the process returns to block 312, where the system 300 performs model training using a different feature set of the training set. In response to determining that one or more trained models have an accuracy that satisfies the threshold accuracy, the process continues to block 316. The system 300 may discard trained machine learning models having an accuracy below the threshold accuracy (e.g., based on the validation set).

[0059] At block 316, the system 300 performs model selection to determine which of the one or more trained models that meet the threshold accuracy has the highest accuracy (e.g., the selected model 308 based on the validation of block 314). In response to determining that two or more trained models that meet the threshold accuracy have the same accuracy, the process can return to block 312, where the system 300 performs model training using a further refined training set corresponding to the further refined feature set to determine the trained model with the highest accuracy.

[0060] At block 318, the system 300 performs model testing using the test set 306 to test the selected model 308. The system 300 may test the first trained machine learning model using the first feature set in the test set to determine that the first trained machine learning model meets the threshold accuracy (e.g., based on the first feature set of the test set 306). In response to the accuracy of the selected model 308 not meeting the threshold accuracy (e.g., the selected model 308 overfits the training set 302 and / or the validation set 304 and is not applicable to other data sets, such as the test set 306), the process continues to block 312, where the system 300 performs model training (e.g., retraining) using a different training set corresponding to a different feature set. In response to determining that the selected model 308 has an accuracy that meets the threshold accuracy based on the test set 306, the process continues to block 320. At least in block 312, the model may learn patterns in the historical images 342 to make predictions, and in block 318, the system 300 may apply the model to the remaining data (e.g., the test set 306) to test the predictions.

[0061] At block 320, the system 300 uses a trained model (e.g., the selected model 308) to receive an input video 348 and corresponding video attributes 344, and extracts a corresponding video classification 346 from the output of the trained model, which represents the predicted quality score of the input video at each possible combination of video format, transcoding configuration, and display resolution tuples. For a single input video, the trained model can be used to predict a per-frame quality score for each sampled frame, and these frame scores can then be aggregated to obtain an overall score for the input video at a particular video format, transcoding configuration, and display resolution tuple. In other forms, the aggregation method can also vary, i.e., average to weighted average, etc.

[0062] The final output is a 3D quality matrix for all possible index tuples (video_format_index, transcoding_config_index, display_resolution_index). Fig. 8A and 8B An example graphical representation of the output of a trained model of an embodiment of the present disclosure is shown. Fig. 8A and 8B Predicted VMAF scores for videos transcoded to 360p, 480p, and 720p versions and displayed at 480p and 720p are shown, where the various scores predicted by the trained model are very close to the corresponding ground truth scores.

[0063] In some embodiments, after a dataset is generated and used to train, validate, or test the machine learning model 160, the machine learning model 160 may be further trained, validated, tested (e.g., using manually determined image classifications, etc.), or adjusted (e.g., adjusting weights associated with the input data of the machine learning model 160, such as connection weights in a neural network).

[0064] In some embodiments, one or more of actions 310-320 may occur in various orders and / or with other actions not presented and described herein. In some embodiments, one or more of actions 310-320 may not be performed. For example, in some embodiments, one or more of the data partitioning of block 310, the model validation of block 314, the model selection of block 316, or the model testing of block 318 may not be performed.

[0065] For performance and latency improvements during embodiments of the present disclosure, the trained model can be combined with temporal sampling of the input video. For example, the system can utilize a few sampled frames of the input video (rather than all frames) as input to the trained model to predict overall quality. Using such temporal sampling of the input video, significant speed improvements can be achieved with little to no loss in accuracy. These speed gains can be achieved without any retraining of the model.

[0066] In embodiments of the present disclosure, the video classification 346 obtained from using the trained model 320 can be used to optimize the video classification 346 on the client side (e.g., in a video such as Figure 1 The format of the media viewer 105 is selected. Fig. 8A In the example shown in , when the display resolution is 480p and the currently playing version is 360p, it is assumed that there is enough bandwidth for all transcoded versions. According to the predicted quality scores, the perceived quality of the 480p and 720p versions is very close, and they are both significantly higher than the 360p version. In this case, the optimal format is the 480p transcoded version, not the 720p.

[0067] Without the currently proposed algorithm that provides a prediction score, a media viewer may blindly switch to a 720p version that uses more bandwidth but does not improve the actual viewing experience (e.g., perceived quality measured by, for example, PSNR or VMAF). Figure 8B ), since the difference between 480p and 720p is greater than 6 points on VMAF (which roughly translates to 1 stop higher in noticeable difference), stepping up to 720p improves the actual perceived quality.

[0068] Another advantage of the disclosed embodiments is that it allows accurate quality assessment (e.g., at the CRF level). For example, a media viewer can calculate the exact quality improvement of a 480p display resolution by switching from a 480p CRF 40 version to a 720p CRF 30 version to further optimize format selection.

[0069] Figure 4 A flowchart of an example of a method 400 for training a machine learning model according to one or more aspects of the present disclosure is depicted. The method is performed by processing logic that may include hardware (circuitry, dedicated logic, etc.), software (such as running on a general-purpose computer system or a dedicated machine), or a combination thereof. In one embodiment, by Figure 1 The method is performed by the computer system 100, and in some other embodiments, it can be performed by one or more other machines not depicted in the figure. Figure 4 In some aspects, the Figure 1 The training set generator 131 of the server machine 130 and / or Figure 2 The training set generator 231 performs Figure 4 One or more boxes.

[0070] For simplicity of explanation, method is depicted and described as a series of actions. However, actions according to the present disclosure can occur in various orders and / or simultaneously, and occur together with other actions not presented and described herein. In addition, not all illustrated actions may be required to implement the method according to the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that these methods can be represented as a series of interrelated states via state diagrams or events instead. In addition, it should be understood that the methods disclosed in this specification can be stored on articles of manufacture to facilitate transporting and transferring these methods to computing devices. As used herein, the term article of manufacture is intended to encompass a computer program accessible from any computer-readable device or storage medium.

[0071] Method 400 begins by generating training data for a machine learning model. In some embodiments, in box 401, the training set T is initialized to an empty set. In box 402, a reference video is transcoded to multiple resolution formats with multiple transcoding configurations. In one embodiment, the multiple resolution formats include 360p, 480, 720p, 1080p, and 2160p (4K), etc. The multiple transcoding configurations may include CRFs from 0 to 51 using an H.264 encoder. Other codecs and encoders may be used for transcoding configurations in embodiments of the present disclosure, such as the VP9 codec. In one example, a reference video may be a video with a resolution of 480p, which is transcoded to each resolution format of 360p, 480p, 720p, 1080p, and 2160p in each transcoding configuration of an H.264 encoder from CRF 0 to 51. For example, a 480p reference video can be transcoded to 720p CRF 0 version to 720p CRF 51 version, and so on, for all resolution formats and transcoding configurations, resulting in multiple different transcoded versions of the reference video.

[0072] In box 403, each transcoded version of the reference video generated in box 402 is rescaled to multiple different display resolutions. In one embodiment, the multiple different display resolutions represent possible display resolutions of the client device that plays the transcoded version of the reference video. The multiple different display resolutions may include 360p, 480p, 720p, 1080p, and 2160p, etc. Referring to the above example, taking the 720p CRF 30 version of the 480p reference video as another example, the 720p CRF30 version can be rescaled to each display resolution of 360p, 480p, 720p, 1080p, and 2160p to generate a rescaled transcoded version of the 480p reference video. This rescaling is performed for each transcoded version of the reference video generated in box 402 to generate multiple rescaled transcoded versions of the reference video.

[0073] At block 404, a quality score is obtained for each frame of each rescaled, transcoded version of the reference video. In one embodiment, the quality score may be a PSNR score or a VMAF score determined for the frames of the rescaled, transcoded version of the reference video. PSNR refers to the ratio between the maximum possible power of a signal and the power of corrupting noise that affects the fidelity of its representation. PSNR may be expressed in logarithmic decibel scales. VMAF refers to a full-reference video quality metric that predicts the quality of a subject video based on reference and distorted video sequences.

[0074] In box 405, an input / output mapping is generated for each frame of the reference video. An input / output mapping refers to a training input that is mapped to a specific target output. The training input includes or is based on a set of attributes of the reference video. The target output for the training input identifies the quality score of the frame of the rescaled transcoded version of the reference video. The training input is associated with (or mapped to) the target output. In one embodiment, the attribute set of the reference video includes color attributes (e.g., RGB intensity values ​​or Y intensity values ​​for each frame), spatial attributes (e.g., Gabor filter features for each frame), and temporal attributes (e.g., optical flow for each frame). In one embodiment, the attribute set of each frame of the reference video is mapped to an input for the quality score of each frame of each rescaled transcoded version of the reference video. In box 406, the input / output mapping generated in box 405 is added to the training set T.

[0075] Block 407 branches based on whether the training set T is sufficient for training the machine learning model 160. If so, execution proceeds to block 408, otherwise, execution returns to block 402. It should be noted that in some embodiments, the sufficiency of the training set T may be determined based on the number of input / output mappings in the training set, while in some other embodiments, the sufficiency of the training set T may be determined based on one or more other criteria (e.g., a measure of diversity of training examples, etc.) in addition to or instead of the number of input / output mappings.

[0076] At block 408, a training set T is provided to train the machine learning model 160. In one embodiment, the training set T is provided to the training engine 141 of the server machine 140 to perform training. For example, in the case of a neural network (e.g., CNN), the input values ​​of a given input / output mapping (e.g., pixel values ​​of a training image, etc.) are input to the neural network, and the output values ​​of the input / output mapping are stored in the output nodes of the neural network. The connection weights in the neural network are then adjusted according to a learning algorithm (e.g., back propagation, etc.), and the process is repeated for other input / output mappings in the training set T. After block 408, the machine learning model 160 can then be used to process a video (e.g., according to Figure 5 Method 500, as described below).

[0077] Figure 5 A flowchart of an example of a method 500 for processing a video using a trained machine learning model according to one or more aspects of the present disclosure is depicted. The method is performed by processing logic that may include hardware (circuitry, dedicated logic, etc.), software (e.g., running on a general-purpose computer system or a dedicated machine), or a combination thereof. In one embodiment, using Figure 1The method is performed by the server machine 150 and the trained machine learning model 160, and in some other embodiments, it can be performed by one or more other machines not depicted in the figure. Figure 5 One or more boxes.

[0078] Method 500 may include receiving an input video (e.g., from a user device or a server such as upload server 125) and processing the input video using a trained model such as trained machine learning model 160. The trained model may be configured to generate one or more outputs indicating a predicted quality score for the input video based on pixel data of frames of the input video.

[0079] In some implementations, a video (input video) may be identified for processing at block 501. In one example, the video includes one or more frames of an uploaded video (eg, a video uploaded to a content sharing platform).

[0080] At block 502, a subset of frames is extracted from frames of a video. In one embodiment, one frame out of every 10 frames in the video may be extracted. At block 503, for each extracted frame, a set of attributes of the frame is identified. The set of attributes may include color attributes (e.g., RGB intensity values ​​or Y intensity values ​​for each frame), spatial attributes (e.g., Gabor filter features for each frame), and temporal attributes (e.g., optical flow for each frame).

[0081] At block 504, the attribute set is provided as input to the trained machine learning model. In one embodiment, the trained machine learning model uses Figure 4 The described method 400 is trained. At block 505, one or more outputs are obtained from the trained machine learning model. In one embodiment, the output from the trained machine learning model is a quality score corresponding to each extracted frame of the video. The trained machine learning model generates a plurality of quality scores, each quality score corresponding to all possible index tuples of the video resolution format, transcoding configuration, and display resolution of the input video.

[0082] At block 506, the per-frame quality scores of the extracted frames of the video are combined to generate an overall quality score for the video. An overall quality score is generated for each of the possible index tuples of video resolution format, transcoding configuration, and display resolution. In one embodiment, combining the per-frame quality scores includes aggregating the scores. In some embodiments, the aggregation process can vary and can include averaging to weighted averaging, etc.

[0083] At block 507 , a final output of the trained machine learning model is generated, wherein the final output is a 3D quality matrix for all possible index tuples of video resolution formats, transcoding configurations, and display resolutions.

[0084] Figure 6 A flowchart depicts an example of a method 600 for processing a video using a trained machine learning model and using the output of the trained machine learning model at a server device to optimize format selection in accordance with one or more aspects of the present disclosure. The method is performed by processing logic that may include hardware (circuitry, dedicated logic, etc.), software (such as running on a general purpose computer system or a dedicated machine), or a combination thereof. In one embodiment, the method uses Figure 1 The server machine 150 and the trained machine learning model 160 are executed, and in some other embodiments, Figure 6 One or more blocks of the diagram may be performed by one or more other machines not depicted in the diagram.

[0085] Method 600 may include receiving an input video (e.g., from a user device or a server such as upload server 125) and processing the input video using a trained model such as trained machine learning model 160. The trained model may be configured to generate one or more outputs indicating a predicted quality score for the input video based on pixel data of frames of the input video.

[0086] In some implementations, a video (input video) may be identified for processing at block 601. In one example, the video includes one or more frames of an uploaded video (eg, a video uploaded to a content sharing platform).

[0087] At block 602, a set of attributes of a video is identified. In one embodiment, the set of attributes is determined for each of a subset of frames of the video. The set of attributes may include color attributes (e.g., RGB intensity values ​​or Y intensity values ​​for each frame of the video), spatial attributes (e.g., Gabor filter features for each frame of the video), and temporal attributes (e.g., optical flow for each frame of the video).

[0088] In block 603, the attribute set is provided as input to the trained machine learning model. In one embodiment, the trained machine learning model uses Figure 4The described method 400 is trained. In box 604, one or more outputs are obtained from the trained machine learning model. In one embodiment, the output from the trained machine learning model is a quality score corresponding to each extracted frame of the video. The trained machine learning model generates multiple quality scores, each quality score corresponding to all possible index tuples of the video resolution format, transcoding configuration, and display resolution of the video. In one embodiment, the per-frame quality scores of the extracted frames of the video are combined to generate an overall quality score of the video. An overall quality score is generated for each possible index tuple of the video resolution format, transcoding configuration, and display resolution. In one embodiment, combining the per-frame quality scores includes an aggregate score. In some embodiments, the aggregation process can vary and can include averaging to weighted averaging, etc.

[0089] At block 605, quality score data is extracted from the output obtained at block 604. In one embodiment, the quality score data includes a quality score corresponding to each rescaled, transcoded version of the video. The quality scores each reflect the perceived quality of the video at a corresponding rescaled, transcoded configuration of the video at a particular display resolution. At block 606, the extracted quality scores are used to determine a resolution format to be selected at a media viewer of a client device. In one embodiment, the format analysis engine 152 (at Figure 1 The extracted quality scores may be used by the server machine 150 of the client device 102A and / or by the media viewer 105 of the client device 102A to select a resolution format and a transcoding configuration at a client device having a particular display resolution. The format analysis engine 152 may compare the quality scores to inform the optimal resolution format and transcoding configuration selection for the client device.

[0090] Figure 7 A flowchart depicts an example of a method 700 for optimizing format selection using the output of a trained machine learning model at a client device in accordance with one or more aspects of the present disclosure. The method is performed by processing logic that may include hardware (circuitry, dedicated logic, etc.), software (e.g., running on a general purpose computer system or a dedicated machine), or a combination thereof. In one embodiment, the method uses Figure 1 , and in some other embodiments, Figure 7 One or more blocks of the diagram may be performed by one or more other machines not depicted in the diagram.

[0091] Method 700 may include receiving an input video (e.g., at a client device 102A-10N from a server such as upload server 125) for playback at a media viewer of a client device, where the client device has a specific display resolution, at block 701. The client device performs playback of the video in a current format selection at block 702. The current format selection may include a current video resolution and a current transcoding configuration.

[0092] At block 703, a quality score corresponding to the video is accessed. Using the trained model (e.g., Figure 1 The trained machine learning model 160) generates a quality score. The trained model can be configured to generate one or more outputs based on pixel data of frames of the input video, the one or more outputs indicating the predicted quality score of the input video at various tuples of video resolution, transcoding configuration, and display resolution. The output can be maintained at a server device (such as server machine 150), or can be provided as metadata with the video when the video is received for playback. In one embodiment, the trained machine learning model can be used according to the above Figure 4 Method 400 training.

[0093] At decision block 704, a determination is made as to whether the quality score indicates a perceived improvement at a different format selection (e.g., at a different video resolution and / or transcoding configuration) than the current format selection for the particular display resolution of the client device. Fig. 8A and 8B An example of using a quality score of a trained machine learning model of an embodiment of the present disclosure to optimize format selection at a client device is provided. If the quality score indicates a perceptual improvement under different format selections, method 700 proceeds to box 705, where the current format selection for video playback is changed to a format selection (e.g., video resolution and / or transcoding configuration) that corresponds to a quality score that provides a perceptual quality improvement for the video at the display resolution of the client. Method 700 then returns to box 702 to continue playing the video in the new format selection. At decision box 704, if the quality score does not indicate a perceptual quality improvement under different format selections, method 700 returns to box 702 to continue playing the video in the current format selection.

[0094] Fig. 8A and 8B An example graphical representation of the output of a trained machine learning model for optimizing format selection according to an embodiment of the present disclosure is provided. Fig. 8A and 8BPredicted VMAF scores for videos transcoded to 360p, 480p, and 720p versions and displayed at 480p and 720p are shown, where the various scores predicted by the trained model are very close to the corresponding ground truth scores.

[0095] for Fig. 8A The example shown in , which shows the predicted quality scores (e.g., VMAF) when the display resolution is 480p and the currently playing version is 360p, assuming that bandwidth is sufficient for all transcoded versions. According to the predicted quality scores, the perceptual quality of the 480p and 720p versions is very close, and they are both significantly higher than the 360p version. In this case, the optimal format is the 480p transcoded version, not the 720p.

[0096] Without an embodiment of the present disclosure providing a prediction score, the media viewer blindly moves up to a 720p version that uses more bandwidth but does not improve the actual viewing experience (e.g., perceived quality) of the video. Figure 8B ), since the difference between 480p and 720p is greater than 6 points on VMAF (which roughly translates to 1 stop higher in a just-noticeable difference), upscaling to 720p improves the actual perceived quality of the video. Therefore, in this case, the media viewer should upscale to the 720p version to improve the perceived quality of the video during playback.

[0097] Another advantage of the disclosed embodiments is that it allows accurate quality assessment (CRF level). Fig. 8A and 8B As shown, the media viewer is able to calculate the exact quality improvement for a 480p display resolution from switching from the 480p CRF40 version to the 720p CRF 30 version to further optimize format selection.

[0098] Fig. 9 A block diagram of an illustrative computer system 900 that operates in accordance with one or more aspects of the present disclosure is depicted. In various illustrative examples, computer system 900 may correspond to Figure 1The computer system 900 is a computing device within the system architecture 100. In some embodiments, the computer system 900 can be connected (e.g., via a network 630 such as a local area network (LAN), an intranet, an extranet, and the Internet) to other computer systems. The computer system 900 can operate as a server or client computer in a client-server environment, or as a peer computer in a peer-to-peer or distributed network environment. The computer system 900 can be provided by the following parts: a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a network appliance, a server, a network router, a switch or a bridge, or any device capable of executing a set of instructions (sequentially or otherwise) that specify actions to be taken by the device. In addition, the term "computer" should include a collection of any computers that execute one (or more) sets of instructions alone or in combination to perform any one or more methods described herein.

[0099] On the other hand, the computer system 900 may include a processing device 902, a volatile memory 904 (e.g., a random access memory (RAM)), a non-volatile memory 906 (e.g., a read-only memory (ROM) or an electrically erasable programmable read-only memory (EEPROM)), and a data storage device 916, which may communicate with each other via a bus 908.

[0100] The processing device 902 may be provided by one or more processors, such as a general purpose processor (such as, for example, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a microprocessor that implements other types of instruction sets, or a microprocessor that implements a combination of instruction set types) or a special purpose processor (e.g., an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), or a network processor).

[0101] The computer system 900 may further include a network interface device 922. The computer system 900 may also include a video display unit 910 (eg, LCD), an alphanumeric input device 912 (eg, keyboard), a cursor control device 914 (eg, mouse), and a signal generating device 920.

[0102] The data storage device 916 may include a transitory or non-transitory computer-readable storage medium 924 on which instructions 926 may be stored, the instructions 926 encoding any one or more of the methods or functions described herein, including instructions for implementing the respective Figures 4 to 7 Instructions of methods 400-700.

[0103] The instructions 926 may also reside, completely or partially, within the volatile memory 904 and / or the processing device 902 during execution thereof by the computer system 900, and thus the volatile memory 904 and the processing device 902 may also constitute machine-readable storage media.

[0104] Although the computer-readable storage medium 924 is shown as a single medium in the illustrative example, the term "computer-readable storage medium" shall include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store one or more executable instruction sets. The term "computer-readable storage medium" shall also include any tangible medium that can store or encode a set of instructions for execution by a computer, which instructions cause the computer to perform any one or more of the methods described herein. The term "computer-readable storage medium" shall include, but is not limited to, solid-state memories, optical media, and magnetic media.

[0105] The methods, components and features described herein can be implemented by discrete hardware components, or can be integrated into the functions of other hardware components, such as ASIC, FPGA, DSP or similar devices. In addition, the methods, components and features can be implemented by component modules or functional circuits in hardware devices. In addition, the methods, components and features can be implemented in any combination of hardware devices and computer program components or in a computer program.

[0106] Unless expressly stated otherwise, terms such as "generate", "provide", or "train", etc. refer to actions and processes performed or implemented by a computer system that manipulates and transforms data represented as physical (electronic) quantities within a computer system register or memory into other data that is similarly represented as physical quantities within the computer system memory or register or other such information storage, transmission, or display device. In addition, the terms "first", "second", "third", "fourth", etc., as used herein, are intended as labels for distinguishing different elements and may not have an ordinal meaning according to their numerical designation.

[0107] The examples described herein also relate to an apparatus for performing the methods described herein. The apparatus may be specially constructed for performing the methods described herein, or it may include a general-purpose computer system selectively programmed by a computer program stored in the computer system. Such a computer program may be stored in a computer-readable tangible storage medium.

[0108] The methods and illustrative examples described herein are not inherently associated with any particular computer or other device. Various general purpose systems may be used in accordance with the teachings described herein, or it may prove convenient to construct more specialized devices to perform each of the methods 400 to 700 and / or their respective functions, routines, subroutines, or operations. Structural examples of various such systems are set forth in the above description.

[0109] The above description is intended to be illustrative rather than restrictive. Although the present disclosure has been described with reference to specific illustrative examples and embodiments, it will be appreciated that the present disclosure is not limited to the described examples and embodiments. The scope of the present disclosure should be determined with reference to the appended claims and the full scope of equivalents to which the claims are entitled.

Claims

1. A method for format selection, comprising: Generating training data for a machine learning model to be trained to identify quality scores of a set of transcoded versions of a new video at a set of display resolutions, wherein each display resolution is a resolution at which the transcoded version is displayed after rescaling, wherein generating the training data comprises: generating a plurality of reference transcoded versions of a reference video, wherein the plurality of reference transcoded versions include the reference video transcoded into a plurality of different video resolutions, having a plurality of transcoding configurations, and rescaled to a plurality of display resolutions; Obtaining quality scores of frames of the plurality of reference transcoded versions of the reference video; generating a first training input comprising a set of color attributes, spatial attributes, and temporal attributes of the frame of the reference video; and generating a first target output for the first training input, wherein the first target output comprises the quality scores of the frames of the plurality of reference transcoded versions of the reference video; and providing the training data to train the machine learning model on: (i) a training input set including the first training input and (ii) a target output set including the first target output, Wherein, after training, the machine learning model is configured to process the new video and generate one or more outputs, wherein the one or more outputs indicate a quality score of the new video for each of multiple possible tuples of video resolution, transcoding configuration, and display resolution before rescaling.

2. The method according to claim 1, wherein: The quality score comprises a peak signal-to-noise ratio (PSNR) of the frame.

3. The method according to claim 1, wherein: The quality score comprises a Video Multi-Method Assessment Fusion (VMAF) of the frame.

4. The method according to claim 1, wherein: The color attribute includes at least one of an RGB or Y value of the frame.

5. The method according to claim 1, wherein: The spatial properties include Gabor feature filter banks.

6. The method according to claim 1, wherein: The temporal attributes include optical flow.

7. The method according to claim 1, wherein: The machine learning model is configured to process the new video and generate one or more outputs indicating quality scores of the set of transcoded versions of the new video at the set of display resolutions.

8. The method according to claim 1, wherein: The machine learning model is a convolutional neural network (CNN) training model.

9. An apparatus for format selection, comprising: Memory for storing video; as well as a processing device operatively coupled to the memory for: Providing a sample frame of the video as an input to a trained machine learning model, wherein the trained machine learning model is trained according to any one of claims 1 to 8; obtaining one or more outputs from the trained machine learning model; and A quality score is extracted from the one or more outputs, the quality score indicating a perceived quality of the video in a first video format, a first transcoding configuration, and a first display resolution.

10. The device according to claim 9, wherein: The processing device is further configured to provide the quality score to a client device, the quality score informing a format selection of the video at a media viewer of the client device, wherein the first display resolution is a display resolution of the client device.

11. The device according to claim 10, wherein: The format selection of the video at the media viewer is based on whether a difference between the quality score of the one or more outputs and another quality score exceeds a threshold, the other quality score indicating the perceived quality at a second video format, a second transcoding configuration, and the first display resolution.

12. The device according to claim 9, wherein: The quality score includes at least one of a peak signal-to-noise ratio (PSNR) measurement or a video multi-method assessment fusion (VMAF) measurement.

13. The device according to claim 9, wherein: The trained machine learning model is trained using an input-output mapping comprising an input and an output, wherein the input is based on a set of color attributes, spatial attributes, and temporal attributes of frames of a reference video, and the output is based on quality scores of frames of multiple transcoded versions of the reference video.

14. The device according to claim 13, wherein: The color attribute comprises at least one of an RGB or Y value of the frame, the spatial attribute comprises a Gabor feature filter bank of the frame, and the temporal attribute comprises an optical flow of the frame.

15. The device according to claim 9, wherein: The one or more outputs from the trained machine learning model include a plurality of quality scores corresponding to a plurality of possible index tuples of video formats, transcoding configurations, and display resolutions.

16. A non-transitory machine-readable storage medium storing instructions that, when executed, cause a processing device to perform operations comprising: Generating training data for a machine learning model to be trained to identify quality scores of a set of transcoded versions of a new video at a set of display resolutions, wherein each display resolution is a resolution at which the transcoded version is displayed after rescaling, wherein generating the training data comprises: generating a plurality of reference transcoded versions of a reference video, wherein the plurality of reference transcoded versions include the reference video transcoded into a plurality of different video resolutions, having a plurality of transcoding configurations, and rescaled to a plurality of display resolutions; Obtaining quality scores of frames of the plurality of reference transcoded versions of the reference video; generating a first training input comprising a set of color attributes, spatial attributes, and temporal attributes of the frame of the reference video; and generating a first target output for the first training input, wherein the first target output comprises the quality scores of the frames of the plurality of reference transcoded versions of the reference video; and providing the training data to train the machine learning model on: (i) a training input set including the first training input and (ii) a target output set including the first target output, Wherein, after training, the machine learning model is configured to process the new video and generate one or more outputs, wherein the one or more outputs indicate a quality score of the new video for each of multiple possible tuples of video resolution, transcoding configuration, and display resolution before rescaling.

17. The non-transitory machine-readable storage medium of claim 16, wherein: The quality score includes at least one of a peak signal-to-noise ratio (PSNR) of the frame or a video multi-method assessment fusion (VMAF) of the frame.

18. The non-transitory machine-readable storage medium of claim 16, wherein: The color attribute comprises at least one of RGB or Y values ​​of the frame, and wherein the spatial attribute comprises a Gabor feature filter bank, and wherein the temporal attribute comprises optical flow.

19. The non-transitory machine-readable storage medium of claim 16, wherein: The machine learning model is a convolutional neural network (CNN) training model.

20. The non-transitory machine-readable storage medium of claim 16, wherein: The machine learning model is configured to process the new video and generate one or more outputs indicating quality scores of the set of transcoded versions of the new video at the set of display resolutions.

Images