Encoding video data using machine learning techniques to predict image quality

Abstract

A computer-based method for encoding video data based on predicted quality values ​​generated by machine learning, comprising: providing a target video data package to a neural network to generate a plurality of predicted quality values ​​for the target video data package, each of the plurality of predicted quality values ​​being associated with a different set of target coding parameters from a range of coding parameters, the neural network being trained with training data comprising a plurality of reference video data packages and reference quality values ​​calculated for each reference video data package encoded according to a different set of reference coding parameters from the range of coding parameters; setting target coding parameters for the target video data package based on the plurality of predicted quality values; and transmitting control signals to an encoder module to encode the target video data package using the target coding parameters.

Description

[Technical Field]

[0001] (Cross-reference to related applications) This application claims priority to U.S. Provisional Patent Application No. 63 / 509,585, filed on 22 June 2023, and European Application No. 23190618.1, filed on 9 August 2023. These applications are incorporated in their entirety.

[0002] This disclosure relates to image analysis, and more specifically, to image analysis techniques implemented by machine learning for video data. Such analysis may be used to set one or more encoding parameters for video data. [Background technology]

[0003] Encoding or compressing video data offers various technical advantages. For example, encoded video data tends to be significantly smaller than the source video data. In addition to reducing storage capacity, reducing the size of video data facilitates faster buffering and streaming of video data to end users, lowers the bandwidth requirements of content delivery systems, and improves compatibility between devices and platforms. Furthermore, smaller video data packages reduce the computing power required for processing and transmission, thus reducing the computational demands on data centers and / or content delivery networks, resulting in significant energy savings. [Overview of the Initiative]

[0004] While encoding video data offers many advantages, lossy compression algorithms often remove or reduce some of the visual information. Lossy compression algorithms can introduce artifacts and blurring into encoded video data, particularly when low bitrates or aggressive compression settings are used. Furthermore, video encoding can be computationally intensive and time-consuming. Video encoding often requires users to select from a variety of complex encoding parameters that affect quality, file size, and compatibility. Due to the computational complexity and time-consuming nature of this process, selecting encoding parameters through iterative trial and error is often inefficient.

[0005] Furthermore, to quantify the loss of quality in encoded video data, video quality values ​​can be calculated, but these values ​​typically require comparing the encoded video data to the original source video data. Therefore, such quality values ​​are generated only after (i) the original (i.e., source) unencoded source video data has been encoded and then decoded, and (ii) additional computational processing has been performed to compare the source video data with the encoded video data. Since the computational processing required to analyze and compare the unencoded and encoded video data often has a computational load equivalent to the encoding process itself, applying video quality values ​​to estimate video quality by comparing the encoded video data to the source video data is often more computationally intensive than the encoding process itself. Moreover, because quality values ​​depend on the statistics of the content (each video data package, such as a data file containing a complete piece of content or other related content), quality values ​​generally need to be generated for each different piece of content (each different video data package) and / or regenerated if the encoding parameters used to encode the video data are changed. In some examples, each video data package may contain a collection of video data related to a single piece of video content.

[0006] Given the enormous computational demands associated with (i) encoding video data and (ii) analyzing the encoded data against its unencoded version, performing real-time or near-real-time video quality analysis on encoded video data may be computationally impossible. Therefore, there is a need for optimized image analysis techniques that can dramatically improve the computational throughput of computer systems that generate video encoded at a specific visual quality. Such novel and innovative high-throughput computing techniques rapidly generate encoded streams to have a predicted video quality and / or to have encoding parameters set in real-time or near-real-time (e.g., in a software module such as a video rate distortion controller) for specific content based on the predicted video quality. This significantly reduces the overall computational load and improves the throughput of video data analysis and encoding processes.

[0007] Various aspects of this disclosure relate to high-throughput systems and methods for analyzing source video data and generating predicted video quality values ​​across a range (envelope) of coding parameters, where the coding parameters can be set based on such predictions. In some examples, the coding parameter envelope may include one or more ranges of one or more types of coding parameters.

[0008] For example, a computer-based method for encoding video data based on predicted quality values ​​generated by machine learning includes providing a target video data package to a neural network to generate a plurality of predicted quality values ​​for the target video data package, each of which is associated with a different set of target coding parameters from a range of coding parameters, and the neural network is trained using training data including a plurality of reference video data packages and reference quality values ​​calculated for each reference video data package encoded according to a different set of reference coding parameters from the range of coding parameters; setting target coding parameters for the target video data package based on the plurality of predicted quality values; and sending control signals to an encoder module to encode the target video data package using the target coding parameters.

[0009] In other features, the plurality of predicted quality values ​​include a first surface defining a first predicted quality value as a function of coding parameters, and a second surface defining a second predicted quality value as a function of coding parameters. Setting the target coding parameters of the target video data package based on the plurality of predicted quality values ​​includes selecting coding parameters that maximize the first predicted quality value while satisfying the quality threshold of the second predicted quality value. In other features, the predicted quality values ​​include at least one of a peak signal-to-noise ratio (PSNR) index, a structural similarity index measure (SSIM) index, a multi-scale structural similarity index measure (MS-SSIM) index, a Video Quality Metric (VQM) index, and / or a Video Multi-Method Assessment Fusion (VMAF) score, and bitrate.

[0010] In other features, the target coding parameters include at least one of the coding codec, quantization parameter (QP), output bitrate, and output resolution. In other features, the plurality of predicted quality values ​​are expressed as coefficients of a predefined parametric equation. In other features, the plurality of predicted quality values ​​include an independent variable of the parametric equation, the independent variable representing the coding parameters. In other features, the plurality of predicted quality values ​​include a dependent variable of the parametric equation, the dependent variable representing the predicted quality values. In other features, the parametric equation defines a differentiable curve. In other features, the parametric equation defines a differentiable surface. In other features, the method includes generating the training data by coding each of the plurality of reference video data packages with each different set of coding parameters.

[0011] In other features, the method includes providing metadata of the target video data package to the machine learning model, the metadata including at least one of the file type, selected target encoding parameters, intended playback software, and intended playback device type. In other features, the method includes calculating an actual quality value by comparing the encoded target video data package to the target video data package, and retraining the machine learning model using the actual quality value. In other features, the machine learning model includes a first machine learning model configured to generate a plurality of predicted first quality values, and a second machine learning model configured to generate a plurality of predicted second quality values.

[0012] In some embodiments, the system includes memory hardware configured to store instructions and processor hardware configured to execute the instructions. The instructions include providing a target video data package to a neural network to generate a plurality of predictive quality values ​​for the target video data package, each of which predictive quality values ​​is associated with a different set of target coding parameters from a range of coding parameters, and the neural network is trained using training data comprising a plurality of reference video data packages and reference quality values ​​calculated for each reference video data package coded according to a different set of reference coding parameters from the range of coding parameters; setting target coding parameters for the target video data package based on the plurality of predictive quality values; and sending a control signal to an encoder module to encode the target video data package using the target coding parameters.

[0013] Other examples, embodiments, features, and aspects will become apparent from the detailed description and accompanying drawings. [Brief explanation of the drawing]

[0014] Figure 1 is a block diagram of an example implementation of a system that implements machine learning techniques to analyze a source video data package and generate predicted video quality values ​​across an envelope of encoding parameters.

[0015] Figure 2 is a flowchart illustrating an example process of training a machine learning model, using the trained machine learning model to analyze a source video data package, and generating predicted video quality for an arbitrary given set of encoding parameters or across the entire envelope of encoding parameters.

[0016] Figures 3A and 3B are flowcharts illustrating an example of the process for generating training data suitable for training a machine learning model.

[0017] FIG. 4 is a flowchart of an example process for calculating an objective quality value of encoded video data.

[0018] FIG. 5 is a flowchart showing an example process for calculating a quality index value of structural similarity index measurement for encoded video data.

[0019] FIG. 6 is a flowchart showing an example process for converting video data into a video tensor suitable for input to a neural network.

[0020] FIGS. 7A to 7B are flowcharts of an example process for training a machine learning model.

[0021] FIGS. 8 and 9 show examples of predicted quality values generated by a trained machine learning model.

[0022] In the drawings, reference numbers may be reused to identify similar and / or identical elements.

Mode for Carrying Out the Invention

[0023] Figure 1 is a block diagram showing an example implementation of System 100, which includes a machine learning platform 102 and an image processing platform 104. As will be described later in this specification, System 100 analyzes source video data (e.g., unencoded video data and / or video data encoded at very high bitrate and / or quality settings [e.g., mezzanine video files encoded using the JPEG2000 standard]) and generates predicted image quality values ​​across an envelope of encoding parameters. For example, the envelope of encoding parameters may include one or more ranges of one or more types of encoding parameters. The predicted image quality values ​​are used to set the encoding parameters for the source video data (e.g., in a rate distortion controller). As shown in Figure 1, System 100 may include a communication system 106. The platforms of System 100 (such as the machine learning platform 102 and the image processing platform 104) may communicate via the communication system 106. An example of the communication system 106 may include one or more networks.That is, GPRS (General Packet Radio Service) networks, TDMA (Time-Division Multiple Access) networks, CDMA (Code-Division Multiple Access) networks, GSM (Global System of Mobile Communications) networks, EDGE (Enhanced Data Rates for GSM Evolution) networks, HSPA (High-Speed ​​Packet Access) networks, HSPA+ (Evolved High-Speed ​​Packet Access) networks, LTE (Long Term Evolution) networks, WiMAX (Worldwide Interoperability for Microwave Access) networks, 5G (5th-Generation) mobile networks, Internet Protocol (IP) networks, Wireless Application Protocol (WAP) networks, or IEEE 802.11 standard networks, as well as any appropriate combination of the above networks. In various implementations, the communication system 106 may also include optical networks, local area networks, and / or global communication networks such as the Internet.

[0024] In some examples, the machine learning platform 102 includes a shared system resource 108, a communication interface 110, and one or more data stores, including non-temporary computer-readable storage media such as storage 112. In some implementations, the shared system resource 108 includes one or more electronic processors, one or more graphics processing units, volatile computer memory, non-volatile computer memory, and / or one or more system buses connecting the components of the shared system resource 108, the communication interface 110, and / or storage 112. In various implementations, storage 112 includes one or more software modules, such as a machine learning training module 120 and / or machine learning module 122. Additional features of the machine learning training module 120 and machine learning module 122 are described later in this specification with reference to Figures 2 to 7B.

[0025] In some embodiments, the video processing platform 104 includes a shared system resource 114, a communication interface 116, and one or more data stores including non-temporary computer-readable storage media such as storage 118. In some examples, the shared system resource 114 includes one or more electronic processors, one or more graphics processing units, volatile computer memory, non-volatile computer memory, and / or one or more system buses connecting the components of the shared system resource 114, the communication interface 116, and / or storage 118. In various implementations, storage 118 includes one or more software modules, for example, a video coding module 124, a video analysis module 126, and / or a rate distortion controller module 128. Additional functions of the video coding module 124, the video analysis module 126, and / or the rate distortion controller module 128 are described later herein with reference to Figures 2 to 7B.

[0026] In various implementations, the components of the machine learning platform 102 communicate with the components of the video processing platform 104 via the communication system 106. For example, the components of the machine learning platform 102 communicate with the communication interface 110, and the components of the video processing platform 104 communicate with the communication interface 116. The communication interface 110 and the communication interface 116 can then communicate with each other via the communication system 106.

[0027] In Figure 1, the machine learning platform 102 and the video processing platform 104 are shown as two separate platforms, but the machine learning platform 102 and the video processing platform 104 may be integrated into the same platform. For example, shared system resource 108 and shared system resource 114 may contain the same components. Similarly, storage 112 and storage 112 may contain the same components. In some implementations, the machine learning platform 102 and the video processing platform 104 may communicate via dedicated wired and / or wireless connections. In some examples, the machine learning platform 102 may be implemented as a subcomponent of the video processing platform 104.

[0028] Figure 2 is a flowchart of an example process 200 that (i) trains a machine learning model, (ii) analyzes source video data using the trained machine learning model, and (iii) generates predicted image quality values ​​for an arbitrary given set of encoding parameters or for the entire envelope of encoding parameters. It should be understood that the training process described herein is optional and may be performed by a separate process. In such embodiments, process 200 would include only the analysis and value generation described herein. Process 200 is described herein as being performed by machine learning platform 102. However, it should be understood that process 200 or any part thereof may also be performed by a platform or a set of devices.

[0029] In step 202, the machine learning training module 120 generates training data. The machine learning training module 120 may generate training data from a library of reference video data (e.g., reference video data stored in storage 112 and / or storage 118). In some examples, the training data includes source video data as reference video data. The training data may also include, for each reference video data package (e.g., video data associated with each content): (i) one or more different sets of encoding parameters, (ii) a reference quality value calculated for each reference video data package encoded according to each of the different sets of encoding parameters, and / or (iii) one or more reference video tensors suitable for input to the machine learning model. In some embodiments, the encoding parameters include parameters of different classes. For example, the encoding parameters may include the encoding codec, quantization parameter (QP) for the encoder, output bitrate, output resolution, output frame rate, output audio parameters, constant rate factor (CRF) configuration, group of picture (GOP) length, and / or buffer size. In some examples, one or more encoding parameters include a combination of parameters of a single class. In other examples, one or more coding parameters include combinations of parameters from multiple classes.

[0030] For example, different sets of coding parameters can include different combinations of a first coding parameter (such as a QP value) and a second coding parameter (such as a bitrate value) across the envelope of coding parameters. As a non-limiting example, if coding parameters include n unique first coding parameters and m unique second coding parameters, different sets of coding parameters can include n × m unique combinations of the first and second coding parameters. In some examples, the unique combinations of the first and second coding parameters are stored in an n × m array or matrix. In some examples, the first coding parameters include QP values ​​in the range of about 16 to about 32. In other examples, the first coding parameters include QP values ​​in the range of about 0 to about 20. In various implementations, the first coding parameters include QP values ​​in the range of about 20 to about 30. In some embodiments, the first coding parameters include QP values ​​in the range of about 30 to about 40. In an example of the embodiment, the first coding parameter includes a QP value in the range of about 40 to about 51.

[0031] In various implementations, the second encoding parameter includes an output resolution ranging from approximately 320 vertical lines to approximately 3840 vertical lines. In some examples, the second encoding parameter includes an output bitrate value ranging from approximately 15 megabits per second (Mbps) to approximately 40 Mbps. In an example embodiment, the second encoding parameter includes an output bitrate value ranging from approximately 8 Mbps to approximately 12 Mbps. In other examples, the second encoding parameter includes an output bitrate value ranging from approximately 2 Mbps to approximately 8 Mbps. In various implementations, the second encoding parameter includes an output bitrate value ranging from approximately 300 kilobits per second (Kbps) to approximately 2 Mbps. In some embodiments, the encoder codec may be specified as a High Efficiency Video Coding (HEVC) Main10 codec. For example, the encoder codec may be an X.264 codec for generating a digital video stream in the H.264 / MPEG-4 AVC video coding format. In other examples, the encoder codec may be an X.265 codec for creating a digital video stream in the HEVC / H.265 video compression format.

[0032] Reference quality values ​​may include values ​​used to evaluate the performance and effectiveness of video codecs and encoding settings by quantifying, for example, how well encoded video data retains the visual quality of the original source video data while minimizing artifacts and maintaining an acceptable file size or bitrate. Examples of quality values ​​include objective metrics such as the peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), multi-scale structural similarity index measure (MS-SSIM), video quality metric (VQM), and / or video multi-method assessment fusion (VMAF) score. In various implementations, one or more reference quality values ​​may be used for each reference video data package. For each video data package, the reference quality values ​​may include reference quality values ​​corresponding to each unique combination of encoding parameters (each different set of encoding parameters). For example, a reference quality value for a given combination of encoding parameters may indicate how well a reference video data package encoded with a particular combination of encoding parameters performs relative to the corresponding source video data package. In various implementations, the reference video data is preprocessed and converted into a format suitable for input to a machine learning model (e.g., a neural network). For example, each reference video data package is preprocessed and converted into a corresponding reference video vector, array, and / or tensor(s). Additional details related to training data generation in 202 are explained with reference to Figures 3A to 3B.

[0033] In 204, the machine learning training module 120 trains a machine learning model using training data (such as the training data generated in 202). In some examples, the machine learning model includes a neural network such as a convolutional neural network (CNN). Suitable examples of neural networks include 3D-CNN, convolutional long short-term memory recurrent (ConvLSTM), two-stream convolutional network, long-term recurrent convolutional network (LRCN), time-distributed layer-based network, and / or transformer-based network. In various implementations, the trained machine learning model is stored in storage 112 and / or storage 118. Additional details related to training the machine learning model with training data are explained with reference to Figures 7A to 7B.

[0034] In step 206, the machine learning module 122 generates predictive quality values ​​for each different set of encoding parameters by providing the trained machine learning model with (i) the target video data package and optionally (ii) one or more different sets of encoding parameters (such as any of the aforementioned encoding parameters). The target video data package and the optional encoding parameters can be entered or selected by the user on a graphical user interface. This is done, for example, through a graphical user interface element that is generated and output to the display of the machine learning platform 102, the video processing platform 104, and / or a separate user device (such as a workstation, terminal, mobile phone, tablet, or other computing device) connected to the machine learning platform 102 and / or the video processing platform 104. In various implementations, the user selects a user interface element corresponding to the target video data package, and depending on the user's selection, the machine learning platform 102 retrieves the target video data package from one or more data stores (such as storage 112 and / or storage 118).

[0035] In some embodiments, the generated predictive quality values ​​may be output to a graphical user interface. In various implementations, the coding parameters may include any one of the classes of coding parameters contained in the training data used to train the machine learning model. In some examples, the trained machine learning model may generate predictive quality values ​​for each combination of coding parameters (each different set of coding parameters). In an example of an embodiment, the trained machine learning model may generate predictive quality values ​​that indicate the predictive quality evaluation of the target video data package when the target video data package is coded using a particular combination (set) of coding parameters. For example, if the coding parameters provided in 206 include n first coding parameters and m second coding parameters, the machine learning model may generate n × m predictive quality values ​​(such as one or more predictive quality values ​​correlated to each combination of coding parameters).

[0036] In various implementations, predictive quality values ​​can be represented as an n×m array. For example, predictive values ​​from various quality values ​​can be stored as an additional dimension in each combination (n,m) of encoded parameters. In some embodiments, predictive quality values ​​can be generated for randomly sampled combinations of encoded parameters (n(random sample), m(random sample)). In other examples, predictive quality values ​​can be represented in parametric form. For example, they can be represented as equations defining differentiable curves or differentiable surfaces. If the predictive quality values ​​include a single class of predictive quality values ​​for the entire range of a single class of encoded parameters, the predictive quality values ​​are represented as equations defining differentiable curves. If the predictive quality values ​​include a single class of predictive quality values ​​for the entire range of two different classes of encoded parameters, the predictive quality values ​​are represented as equations defining differentiable surfaces. In various examples, a trained machine learning model may output one or more parametric equations. In some implementations, a trained machine learning model may output the coefficients of one or more parametric equations.

[0037] Equation (1) below shows a function F(x) that outputs a predicted quality value F for any given input coding parameter x. F(x) = a·f(x) + b (1)

[0038] In equation (1), F(x) can define a differentiable curve. The function f(x) represents a predefined function. The coefficients a and b are generated by a trained machine learning model. Therefore, after the trained machine learning model outputs the coefficients a and b, the user can input an arbitrary coding parameter x into equation (1) and quickly calculate the prediction quality value F for the given coding parameter x.

[0039] Equation (2) below shows a function G(x,y) that outputs a predicted quality value G for any given combination of input coding parameters (x,y). G(x,y)=c·g(x,y)+d (2)

[0040] In equation (2), G(x,y) can define a differentiable surface. The function g(x,y) represents a predefined function. The coefficients c and d are generated by a trained machine learning model. Similar to equation (1), after the trained machine learning model outputs the coefficients c and d, the user can input any combination of coding parameters (x,y) into equation (1) and quickly calculate the predictive quality value G for a given combination of coding parameters (x,y). In some examples, if the user knows the desired predictive quality values ​​F and / or G, equations (1) and / or (2) can be solved for the coding parameters (x,y). In various implementations, the concepts described above with respect to equations (1) and (2) can be scaled to include any number and combinations of input coding parameters and any number and combinations of output predictive quality values. For example, a machine learning model may output two predictive quality values ​​G1(x,y) and G2(x,y) for any given combination of input coding parameters (x,y).

[0041] In 208, the machine learning module 122 and / or the rate distortion controller module 128 select one or more predicted quality values ​​based on one or more threshold quality values. For example, the machine learning module 122 and / or the rate distortion controller module 128 may read one or more threshold quality values. A threshold quality value may represent a minimum or target value for a particular quality value. For example, a threshold quality value may include target values ​​for one or more visual quality values ​​(VMAF, SSIM, etc.). In some embodiments, a user can input threshold quality values ​​into an input field of a graphical user interface. The machine learning module 122 and / or the rate distortion controller module 128 parses the predicted quality values ​​to select one or more predicted quality values ​​for the target image. For example, the machine learning module 122 and / or the rate distortion controller module 128 compares the predicted quality values ​​output from a trained machine learning model with one or more threshold quality values ​​to identify one or more predicted quality values ​​that are closest to the target quality value.

[0042] In an example where a machine learning model is configured to output two predicted quality values ​​G1(x,y) and G2(x,y), the machine learning module 122 and / or the rate distortion controller module 128 may be configured to find coding parameters (x,y) that maximize the quality values ​​G1 and G2. For example, the machine learning module 122 and / or the rate distortion controller module 128 sets a first threshold for G1 and finds a combination of coding parameters (x1,y1) that satisfies or exceeds the first threshold. The machine learning module 122 and / or the rate distortion controller module 128 sets a second threshold for G2 and finds a combination of coding parameters (x2,y2) that satisfies or exceeds the second threshold. In some embodiments, the machine learning module 122 and / or the rate distortion controller module 128 finds a combination of coding parameters (x,y) that satisfies or exceeds both the first quality threshold for G1 and the second quality threshold for G2. For example, the machine learning module 122 and / or the rate distortion controller module 128 find combinations of encoding parameters (x,y) that exist in both (x1,y1) and (x2,y2).

[0043] In various implementations, the machine learning module 122 and / or the rate distortion controller module 128 select a combination of coding parameters (x,y) that maximizes a first quality value (G2) while satisfying quality constraints (e.g., a second quality threshold) on a second quality value (G1). For example, the machine learning module 122 and / or the rate distortion controller module 128 select a combination of coding parameters from (x1,y1) that maximizes G1 and lies on a surface described by G2(x1,y1). In some embodiments, the machine learning module 122 and / or the rate distortion controller module 128 select a combination of coding parameters (x,y) that maximizes a second quality value G2 while satisfying quality constraints (e.g., a first quality threshold) on a first quality value G1. For example, the machine learning module 122 and / or the rate distortion controller module 128 select a combination of coding parameters from (x2, y2) that maximizes G2 and lies on a surface described by G1(x2, y2).

[0044] In 210, the machine learning module 122 and / or the rate distortion controller module 128 set the target coding parameters for the target video based on a set of coding parameters associated with one of several selected predicted quality values. For example, the machine learning module 122 and / or the rate distortion controller module 128 set the target coding parameters for the target video based on a set of coding parameters selected in 210. The target coding parameters may include coding parameters associated with the selected predicted quality value, or may be generated according to these parameters. For example, in some embodiments, the coding parameters associated with the selected value may be modified (e.g., scaled) when used to set the target coding parameters for the target video, depending on how close the corresponding value is to a threshold. As mentioned above, the predicted quality value may be expressed as a parametric equation. In such an implementation, the machine learning module 122 and / or the rate distortion controller module 128 determine the target coding values ​​required to achieve the desired target quality value by solving the parametric equation.

[0045] In various implementations, contextual data (such as metadata) may be provided to the machine learning module 122 and / or the rate-distortion controller module 128 (for example, along with the target video data in 206). Examples of metadata include at least one of the file type, selected target encoding parameters, intended playback software, and intended playback device type. The machine learning module 122 can automatically select the target encoding parameters from the predicted quality values ​​based on the metadata.

[0046] The target encoding parameters can be output to a graphical user interface. In 212, the machine learning module 122 and / or the rate-distortion controller module 128 pass the target encoding parameters to the video encoding module 124. The video encoding module 124 encodes the target video using the target encoding parameters, resulting in an encoded video. In various implementations, the encoded video is stored in storage 112 and / or storage 118. In some examples, the machine learning module 122 and / or the video encoding module 124 calculate the actual quality value by comparing the encoded target video data package with the target video data package. Then, the machine learning module 122 retrains the machine learning model using the actual quality value.

[0047] In some embodiments, the first predictive quality value G1 generated in 206 may be a predictive quality Q represented by the following equation (7).

number

[0048] In equation (7) above, Q represents a surface defined as a function of variables x1 and x2. For example, x1 may be a first encoder parameter (such as a quantization parameter), x2 may be a second encoder parameter (such as resolution), and a, b, c, and d may be coefficients predicted by a machine learning model for a given input image.

[0049] In various implementations, the second predicted quality value G2 generated in 206 may be the predicted bitrate B, which is expressed by the following equation (8).

number

[0050] Similar to equation (7), B represents a surface defined as a function of variables x1 and x2, where e, f, g, and h are coefficients predicted by a machine learning model for a given input image. In some examples, in 208, the rate distortion controller module 128 is instructed to select a target quality value by maximizing quality Q at a target bitrate B. For example, since the value of the target bitrate B is defined, equation (8) can be solved for either x1 or x2. In various implementations, the rate distortion controller module 128 first solves for x1 for x2. For example, equation (8) can be expressed as the following quadratic equation (9), and x1 can be solved using a quadratic method.

number

[0051] The Q'(x2) rate distortion controller module 128 then substitutes the solution (for x1) of equation (9) into equation (7) (surface for quality Q). This yields two different equations Q'(x2), each a function of a single variable x2. The rate distortion controller module 128 then finds the maximum value for each of the two different equations Q'(x2). For example, the rate distortion controller module 128 finds the derivative for each of the two different equations Q'(x2). The value of JPEG2026522457000005.jpg106 is calculated, the derivative is set to equal to 0, and x2 is solved. The rate distortion controller module 128 finds the corresponding value of x1 by substituting x2 into the solution of equation (9). The rate distortion controller module 128 then tests equation (7) Q(x1,x2) with all valid combinations of x1 and x2. The rate distortion controller module 128 stores the calculated values ​​of Q for each combination of x1 and x2 in the array Q * Save to [location].

[0052] In 210, the rate distortion controller module 128 is Q * From this, the combination of x1 and x2 corresponding to the best calculated value of Q is selected. In step 212, the rate distortion controller module 128 encodes the target video data package using the selected combination of x1 (e.g., quantization parameter) and x2 (e.g., resolution). In various implementations, the rate distortion controller module 128 passes the selected combination of x1 and x2 to the video encoding module 124. The video encoding module 124 encodes the target video data package using the selected combination of x1 and x2.

[0053] Figure 2 shows an example where a single machine learning model generates multiple types of predictive quality values, but other implementations may use multiple machine learning models to generate predictive quality values. For example, each of multiple machine learning models may be used to generate a specific type of predictive quality value. Therefore, in an example where two types of predictive quality values ​​are needed, the first machine learning model may generate the first type of predictive quality value, and the second machine learning model may generate the second type of predictive quality value. In various implementations, a single machine learning model may be trained to output multiple types of predictive quality values. For example, a single machine learning model may be trained to output a first type of predictive quality value and a second type of predictive quality value. In some examples, a single machine learning model may be trained to output any number of types of predictive quality values.

[0054] Figures 3A to 3B are flowcharts of an example process 300 for generating training data. In step 302, the machine learning training module 120 may generate combinations of coding parameters. For example, the machine learning training module 120 may select one or more classes of coding parameters (see Figure 2, as described above) and generate random combinations of coding parameters for each class. In step 304, the machine learning training module 120 instructs the video coding module 124 to load multiple reference video data packages. These multiple reference video data packages may be source video data packages selected for training the machine learning model. In step 306, the machine learning training module 120 instructs the video coding module 124 to select the first reference video data package from the multiple reference video data packages. In step 308, the machine learning training module 120 instructs the video coding module 124 to select an initial combination of coding parameters from the coding parameters generated in step 302. In step 310, the machine learning training module 120 instructs the video encoding module 124 to encode the selected reference video data package using the selected combination of encoding parameters. The encoded reference video data package may be stored in storage 112 and / or storage 118.

[0055] In 312, the machine learning training module 120 instructs the video analysis module 126 to generate a reference quality value for the encoded video data package. For example, the encoded video data package is first decoded. During decoding, the encoded video data package is interpreted according to the codec used to encode the video data package, and the video data package is reconstructed. Decoding can be important because the effects of the encoding process (such as loss of detail, blurring, block noise, and / or ringing) may only become apparent after the video data package is reconstructed. After the encoded video data package has been decoded, the machine learning training module 120 generates an objective reference quality value by comparing the decoded video data package with a reference video data package. Additional details related to the generation of the reference quality value are described below herein with reference to Figures 4 to 6. In 314, the machine learning training module 120 associates the generated reference quality value with a selected combination of a reference video data package and selected encoding parameters. The selected reference video data package, the generated reference quality value, and / or the selected coding parameter combination can be stored as training data.

[0056] In step 316, the machine learning training module 120 determines whether there are other combinations of encoding parameters that have not yet been processed in steps 310 through 314. If it determines that there are other unprocessed combinations of encoding parameters ("YES" in decision block 316), the machine learning training module 120 instructs the video encoding module 124 to select the next combination of encoding parameters, and in step 310, encodes the selected video data package using the next selected combination of encoding parameters. If it determines that there are no other unprocessed combinations of encoding parameters ("NO" in decision block 316), the machine learning training module 120 may compile the generated training data and perform any processing necessary to input the training data into a machine learning model. For example, the machine learning training module 120 may convert the selected reference video data package into a reference video tensor(s) suitable for input to a machine learning model. The reference video tensor(s) is associated with the selected reference video data package in the training data. Additional details related to converting a reference video data package into a video tensor are described later herein with reference to Figure 7.

[0057] In step 322, the machine learning training module 120 determines whether there is another reference video data package among the multiple reference video data packages that has not yet been processed in steps 308 through 320. If it is determined that there is another unprocessed reference video data package (YES in decision block 322), the machine learning training module 120 instructs the video encoding module 124 to select the next reference video data package in step 324 and to select the initial combination of encoding parameters again in step 308. If it is determined that there is no other unprocessed reference video data package (NO in decision block 322), the machine learning training module 120 saves the training data. In various implementations, the training data may be stored in storage 112 and / or storage 118.

[0058] If it is determined that another set of unprocessed quality values ​​exists (decision block 336 "YES"), the machine learning training module 120 selects the next set of reference quality values ​​in 338 and again determines in decision block 332 whether the selected reference quality values ​​exceed or are equal to the target quality values. If it is determined that no other set of unprocessed reference quality values ​​exists (decision block 336 "NO"), the machine learning training module 120 converts the selected reference video data package into a reference video tensor (single or multiple) (320). Additional details related to converting the video data package into a video tensor (single or multiple) are described later herein with reference to Figure 6.

[0059] Figure 4 is a flowchart of process example 400 for calculating objective quality values ​​(e.g., Video Multi-method Assessment Fusion (VMAF) score) for encoded video data (e.g., after the encoded video data has been decoded). While process 400 describes the generation of a specific type of quality value, it should be understood that the quality value is not limited to perceived quality values ​​(such as the VMAF score). Rather, any appropriate quality value (such as any combination of those described in process 200) can be used. In 402, the video analysis module 126 loads the encoded video data package. In 404, the video analysis module 126 loads a reference video data package corresponding to the encoded video data package. For example, the video analysis module 126 loads the source video data package used to generate the encoded video data package as the reference video data package. In 406, the video analysis module 126 prepares and aligns the encoded video data package and the reference video data package. For example, the video analysis module 126 can prepare a video data package by analyzing and / or processing the video data package to ensure that the encoded video data package and the reference video data package have the same resolution, frame rate, and / or duration. The video analysis module 126 may also ensure that the encoded video data package and the reference video data package are synchronized by aligning them. In 408, the video analysis module 126 decodes the aligned reference video data package into a first frame sequence. In 410, the video analysis module 126 decodes the aligned encoded video data package into a second frame sequence.

[0060] In 412, the video analysis module 126 calculates key values ​​for each corresponding pair of frames from the first and second frame sequences. In various implementations, key values ​​may include the mean squared error (MSE) metric, the SSIM metric, the visual information fidelity (VIF) metric, the detail loss metric (DLM), and / or the mean co-located pixel difference (MCPD) metric. In 414, the video analysis module 126 normalizes the key values ​​to a common scale. For example, each key metric is normalized to a scale from 0 to 100. In 416, the video analysis module 126 provides the normalized key values ​​to a trained machine learning model to generate a perceptual quality metric for each corresponding pair of frames. In various implementations, the trained machine learning model may be a VMAF model, and the perceptual quality metric may be a VMAF score. In 418, the video analysis module 126 generates an average perceptual quality index for the encoded video data package by averaging the perceptual quality values ​​of each frame pair in the video data package.

[0061] Figure 5 is a flowchart of process example 500 for calculating a structural similarity index quality metric for an encoded video data package. While process 500 illustrates the generation of a specific type of quality value, it should be understood that the quality value is not limited to the structural similarity index metric. Rather, any appropriate quality value (e.g., any combination of those described above for process 200) can be used. In 502, the video analysis module 126 loads the encoded video data package. In 504, the video analysis module 126 loads a reference video data package corresponding to the encoded video data package. For example, the reference video data package may be the unencoded source video data package used to generate the encoded video data package. In 506, the video analysis module 126 ensures that the reference video data package and the encoded video data package are synchronized by aligning them. In 508, the video analysis module 126 extracts corresponding frames from the reference video data package and the encoded video data package. In 510, the video analysis module 126 converts the extracted frames to grayscale. In 512, the video analysis module 126 calculates a measured structural similarity index (SIM) value for each corresponding pair of converted frames. In various implementations, the Structural Similarity Index (SSIM) can be calculated for each pair (x,y) of reference frame x and encoded frame y according to the following equation (3). SSIM(x,y)=l(x,y)·c(x,y)·y(x,y) (3)

[0062] As shown in equation (3) above, the structural similarity index measurement SSIM(x,y) for each pair (x,y) of reference frame x and encoded frame y can be the product of the inter-frame luminance comparison function l(x,y), the inter-frame contrast comparison function c(x,y), and the inter-frame structure comparison function s(x,y). In various implementations, the luminance comparison function l(x,y) for any given pair (x,y) of reference frame x and encoded frame y can be calculated according to the following equation (4).

number

[0063] As shown in equation (4), the luminance comparison function l(x,y) is given by the reference frame μ x Average pixel intensity and encoded frame μ y It can be calculated as the sum of products of the average pixel intensity. In some embodiments, a constant c1 may be added to both the numerator and denominator to avoid instability when the denominator is close to zero.

[0064] In some examples, the contrast comparison function c(x,y) for any given reference frame x and encoded frame y pair (x,y) can be calculated according to equation (5) below.

number

[0065] As shown in equation (5), the contrast comparison function c(x,y) is given by the reference frame σ x The standard deviation of the pixel intensity and the encoded frame σ y It is calculated as the sum of the products of the standard deviation of the pixel intensity. In some embodiments, a constant c2 may be added to both the numerator and denominator to avoid instability that occurs when the denominator is close to zero.

[0066] In some examples, a structure comparison function s(x,y) for any given reference frame x and encoded frame y pair (x,y) can be calculated according to equation (6) below.

Number

[0067] As shown in Equation (6), the contrast comparison function c(x, y) is the covariance σ of the pixel intensities of the reference frame x and the encoded frame y xy divided by the product of the average pixel intensity of the reference frame σ x and the average pixel intensity of the encoded frame σ y and can be calculated as such. In some embodiments, a constant c3 may be added to both the numerator and the denominator to avoid instability when the denominator approaches zero.

[0068] In 514, the video analysis module 126 generates an average structural similarity index measurement index value of the encoded video data package by averaging the structural similarity index measurement index values of each frame pair of the video data package.

[0069] Figure 6 is a flowchart of an example process 600 for generating video tensors suitable for input to a neural network. In 602, the machine learning training module 120, machine learning module 122, video coding module 124, and / or video analysis module 126 load video data. In 604, the machine learning training module 120, machine learning module 122, video coding module 124, and / or video analysis module 126 extract frames from the loaded video data. In 606, the machine learning training module 120, machine learning module 122, video coding module 124, and / or video analysis module 126 rescale the extracted frames to the desired dimensions required by the neural network. In 608, the machine learning training module 120, machine learning module 122, video coding module 124, and / or video analysis module 126 normalize the pixel values ​​within each extracted frame. In some examples, the pixel values ​​may be normalized to a range of approximately 0 to approximately 1. In various implementations, pixel values ​​may be normalized to a range of approximately -1 to approximately 1.

[0070] In 610, the machine learning training module 120, the machine learning module 122, the video coding module 124, and / or the video analysis module 126 convert the processed frames into a multidimensional input matrix (or tensor). In various implementations, the RGB channels of each frame may be separated. This results in three 2D matrices for each frame. In some embodiments, the processed frames may be stacked along the time dimension to create a 3D matrix (height × width × time) or a higher-dimensional matrix (height × width × channels × time). Thus, the video tensor is a spatiotemporal representation of the video data.

[0071] Figures 7A and 7B are flowcharts of an example process 700 for training a machine learning model. In 704, the machine learning training module 120 initializes the machine learning model. In some examples, the machine learning model may include a neural network. The machine learning training module 120 can define the architecture of the neural network (including the number of layers, the number of neurons per layer, and the activation function). The machine learning training module 120 may initialize the weights and biases of the neural network with small random values ​​using an initialization method such as the Xavier method or the He method. In various implementations, the input layer is sized to accept a video tensor of a single-channel (Luma or Y) 4K (3840 x 2160 pixels) video. For example, the dimension of the input layer can be (3840 x 2160 x 3) by concatenating every three adjacent frames of 4K video. In some embodiments, subsequent layers of the neural network are constructed to increase the depth while decreasing the width and height using learnable convolutional filters. In example implementations, the output layer is sized to correspond to an index in a predefined table of possible coding parameters. In other examples, the output layer is sized to directly return the optimal encoding parameters (e.g., scalar values ​​corresponding to QP and resolution).

[0072] In step 704, the machine learning training module 120 loads training data, such as the training data generated in step 202 of process 200, and / or the training data generated according to process 300. In step 706, the machine learning training module 120 divides the training data into multiple batches. In various implementations, a batch contains a predefined number of reference videos (and corresponding reference quality values, coding parameters, and / or video tensors (one or more)). In some examples, a batch contains 32, 64, or 128 reference videos. In various implementations, each video is divided into batches, each batch being sized to contain a specific length of video. For example, each batch may be sized to contain a tensor corresponding to a 2-second video segment.

[0073] In step 708, the machine learning training module 120 selects an initial batch of training data. In step 710, the machine learning training module 120 selects an initial reference video data package within the selected batch. In step 712, the machine learning training module 120 provides the initialized neural network with the reference video tensor and / or coding parameters associated with the selected reference video data package and generates an output. In step 714, the machine learning training module 120 calculates the error between the output and the reference quality value associated with the selected reference video data package. In step 716, the machine learning training module 120 determines whether the endpoint of the selected batch has been reached. If it determines that the endpoint of the selected batch has not been reached (decision block 716 is "NO"), the machine learning training module 120 selects the next reference video data package within the selected batch in step 718 and provides the initialized neural network with the reference video tensor and / or coding parameters associated with the selected reference video data package in step 712. If the machine learning training module 120 determines that it has reached the endpoint of the selected batch (decision block 716 is "YES"), it calculates the average loss of the selected batch in 720 by averaging the errors (e.g., the errors calculated in 714) across all reference video data packages in the selected batch.

[0074] In step 724, the machine learning training module 120 calculates the gradient of the mean loss for the neural network's weights and biases using the chain rule of calculus. In step 726, the machine learning training module 120 updates the neural network's weights and biases using the calculated gradient and a predefined learning rate. In step 728, the machine learning training module 120 determines whether the end of the epoch has been reached. In some examples, the epoch is completed after the data and errors associated with each reference video data package in the training dataset have been calculated. In other words, the epoch is completed after the machine learning training module 120 has completed a cycle of the entire training dataset. If it determines that the end of the epoch has not been reached (decision block 728 is "NO"), the machine learning training module 120 selects the next batch in step 730 and the initial reference video data package in the selected batch in step 710.

[0075] If it is determined that the end of the epoch has been reached (decision block 728: "YES"), the machine learning training module 120 determines whether the training conditions are met. In some examples, the training condition may be whether the mean error between the output and the reference quality value is below a threshold. In various implementations, the threshold may be approximately 5%. In other examples, the training condition is met when a certain number of epochs have been reached. If it is determined that the training conditions are not met (decision block 732: "NO"), the machine learning training module 120 selects the initial batch again at 708. If it is determined that the training conditions are met (decision block 732: "YES"), the machine learning training module 120 saves the neural network as a trained neural network with updated weights and biases.

[0076] Figures 8 and 9 show examples of predicted quality values ​​generated by a trained machine learning model. In the examples in Figures 8 and 9, the predicted quality values ​​are parametric equations that define the surface as a function of resolution and bitrate, respectively.

[0077] The systems and methods described herein offer a variety of novel and original solutions to technical challenges related to digital video analysis systems. For example, as mentioned above, source video data packages (especially high-resolution, high-bitrate data packages) tend to be large. To reduce computational, storage, and / or energy requirements in data centers and / or content distribution systems, it is often desirable to encode video data packages to reduce file size. Encoded video data packages typically have smaller file sizes than raw source video data packages, requiring less storage space (and also tending to improve computational efficiency in distribution and playback). However, as mentioned above, determining optimal encoding parameters can be difficult and computationally intensive. While quality values, as described herein, quantitatively describe the relative quality loss suffered by encoded video data compared to an unencoded source, the process of generating encoded video data objects and calculating quality values ​​for those objects ultimately becomes more computationally intensive than encoding the video data objects in the first place. For example, generating a quality value requires the user to perform a series of computationally intensive operations, including (i) first encoding the source video data object, and (ii) comparing the encoded video data object with the unencoded source video data object. Performing these operations in real time or near real time is often impractical.

[0078] The systems and methods described herein provide a novel and original solution that enables the generation of predicted performance values ​​for a given combination of target video data and encoding parameters (or the entire envelope of predicted performance values ​​generated for the target video data and the entire envelope of encoding parameters) without first encoding the target video data object and performing computationally intensive comparison operations. By dramatically reducing the computational operations required for users to access predicted performance values, the computational throughput of any computer system that calculates performance values ​​can be greatly improved.

[0079] Furthermore, the systems and methods described herein enable the generation of the entire range of predicted quality values ​​for the entire envelope of encoding parameters at once, instead of generating a single set of quality values ​​corresponding to specific encoding parameters used to encode video data. In examples where the range of predicted quality values ​​is generated in parametric form, quality values ​​for combinations of encoding parameters can be extracted with high accuracy (in real time or near real time), even for encoding parameters that were not input into and / or used to train the machine learning model. Moreover, after the range of quality values ​​has been generated (e.g., as individual numerical values ​​or in parametric form), the necessary encoding parameters to achieve any desired quality value can be determined quickly and efficiently in real time or near real time.

[0080] Examples where predicted quality values ​​are expressed by parametric equations defining a perfectly differentiable surface or curve can offer unique technical advantages. For example, the H.265 (High Efficiency Video Coding: HEVC) video compression standard uses algorithms that optimize bitrate, video quality, and encoding complexity. These algorithms are sensitive to small changes in the target bitrate (even small changes in bitrate can cause unpredictable fluctuations in the quality of the encoded video). However, because predicted quality values ​​(such as bitrate) can be generated as a perfectly differentiable surface or curve, local minimums and maximums are eliminated from the predicted quality values ​​when the bitrate is increased or decreased. This reduces unstable responses due to large changes in bitrate and significantly improves the overall stability of implementations where the target video is encoded using the H.265 codec.

[0081] The systems and methods described herein offer diverse technical advantages for various practical applications. For example, a bitrate ladder includes a set of encoded video streams with different bitrates and resolutions, encoded from the same source video data object. Bitrate ladders are used in adaptive streaming techniques to provide optimal video quality for each viewer. Bitrate ladders enable streaming video data with optimal encoding settings for each device. The systems and methods described herein enable simultaneous computation of encoding parameters for the entire bitrate ladder. For example, a machine learning model can generate predicted quality values ​​that define two surfaces. The first surface may be quality as a function of QP and resolution, and the second surface may be bitrate as a function of QP and resolution. To determine the set of encoding parameters to use for a given ladder sequence, an isobitrate curve (which may be all encoding parameters that achieve a target bitrate) can be derived from the second surface. The isobitrate curve can be superimposed on the predicted quality surface, and the highest quality point can be selected. Additionally or alternatively, encoding parameters may be found by solving the intersection of the isobitrate curve and the predicted quality surface. The encoding parameters corresponding to that point can be selected as the target encoding parameters. Since multiple encoding parameters can be found from the surface for multiple target bitrates, this process can be completed simultaneously for multiple target bitrates in the ladder (e.g., 10Mbps, 1Mbps, and / or 100Kbps), and the encoding parameters for the entire bitrate ladder can be derived without repeating the procedure for each target bitrate.

[0082] In various implementations, the video processing platform 104 may be implemented as part of an active encoding platform. The active encoding platform may include a system for processing and encoding digital video data packages into various formats and bitrates for streaming, broadcasting, or storage. The active encoding platform may continuously encode and optimize content in real time or near real time to meet the specific requirements of various playback devices and network conditions. Thus, the video processing platform 104 may continuously generate training data based on the continuously encoded and analyzed video data packages, and the machine learning platform 102 may continuously tune and / or retrain the machine learning model based on the continuously updated training data. In some implementations, some or all of the reference video data packages used to generate training data (e.g., in 202) may be obtained from the active encoding platform rather than from a library of reference video data. In such examples, the source video data, encoded video data, and / or quality values ​​comparing the source video data to the encoded video data are preprocessed and do not need to be generated during the execution of process 200. By eliminating the need to encode video data or generate quality values ​​during the execution of process 200, this implementation further reduces computational requirements and improves computational throughput.

[0083] In an example of the embodiment, training data is generated for N reference video data packages (e.g., by the machine learning training module 120). Each video data package is encoded with M combinations of randomly generated first and second encoding parameters (e.g., quantization parameters and resolution). The encoding parameters may be specified by the user (e.g., via a graphical user interface). For each unique combination of the M randomly generated first and second encoding parameters, (i) the bitrate R of the encoded video data package is R(n i ,mi ) is recorded as (where n i represents a specific first coding parameter, m i (i) represents a specific second encoding parameter, and (ii) the encoded video data package is decoded and analyzed to generate a quality value D, where D(n i ,m i ) is recorded as. After the bitrate R and quality value D are generated for each of the M combinations of the first and second coding parameters, the optimal combination of coding parameters m is determined to achieve the target bitrate r or target quality value d. opt (n) is selected. For example, m to achieve the target bitrate r. opt If (n) is selected, m opt (n) is selected as a set of encoding parameters m that achieve a bitrate less than or equal to the target bitrate r while having the highest quality value d. For example, m is selected to achieve the target quality value d. opt If (n) is selected, m opt (n) is selected as a set of encoding parameters m that achieve a quality value that satisfies the target quality value d or higher while having a minimum bitrate r.

[0084] After the training data is generated, the machine learning model (e.g., a neural network) is trained using backpropagation (e.g., by the machine learning training module 120) to determine the optimal encoding parameter m for each reference video data package in the training data. opt Predict (n). For example, during training, the video tensor corresponding to each reference video data package is input to the machine learning model, and the predictive optimal coding parameters are generated m pred (n). Generated predicted optimal coding parameter m pred (n) is the optimal coding parameter m opt (n) is compared with (from the training dataset for that reference video data package) and the error (e.g., m) is measured. pred (n) and m optThe difference between (n) is generated. Then, the parameters of the machine learning model are adjusted until the error is minimized. During inference (after training), the video tensor corresponding to the target video is input to the machine learning model (for example, in machine learning module 122), and the machine learning model predicts the optimal coding parameters m for the target video. pred (n) is generated. Subsequently, the target video data package is used to predict the optimal coding parameter m pred (n) is used for encoding (for example, in video encoding module 124).

[0085] The following paragraphs provide examples of systems, methods, and apparatus implemented in accordance with this specification.

[0086] Example 1 includes a computer-based method for encoding video data based on predicted quality values ​​generated by machine learning. The method includes providing a target video data package to a neural network to generate a plurality of predicted quality values ​​for the target video data package, each of which is associated with a different set of target coding parameters from a range of coding parameters, and the neural network is trained using training data comprising a plurality of reference video data packages and reference quality values ​​calculated for each reference video data package encoded according to a different set of reference coding parameters from the range of coding parameters; setting target coding parameters for the target video data package based on the plurality of predicted quality values; and sending control signals to an encoder module to encode the target video data package using the target coding parameters.

[0087] Example 2 includes the subject matter of Example 1 and further defines the following: The plurality of predicted quality values ​​include a first surface that defines a first predicted quality value as a function of coding parameters, and a second surface that defines a second predicted quality value as a function of coding parameters, and setting the target coding parameters of the target video data package based on the plurality of predicted quality values ​​includes selecting coding parameters that maximize the first predicted quality value while satisfying the quality threshold of the second predicted quality value.

[0088] Example 3 includes the subject matter of any of Examples 1 or 2 and further defines the following: The predicted quality value includes at least one of the following: peak signal-to-noise ratio (PSNR) index, structural similarity index measure (SSIM) index, multi-scale structural similarity index measure (MS-SSIM) index, video quality metric (VQM) index, and / or video multi-method assessment fusion (VMAF) score, and bitrate.

[0089] Example 4 includes the subject matter of any of Examples 1 to 3 and further defines the following: The target encoding parameters include at least one of the encoding codec, quantization parameter (QP), output bitrate, and output resolution.

[0090] Example 5 includes the subject matter of any of Examples 1, 3, and 4, and further defines the following: The multiple predictive quality values ​​are expressed as coefficients of a predefined parametric equation.

[0091] Example 6 includes the subject matter of Example 5 and further defines the following: The plurality of prediction quality values ​​include the independent variables of the parametric equation, the independent variables representing coding parameters.

[0092] Example 7 includes the subject matter of Example 6 and further defines the following: The plurality of predictive quality values ​​include the dependent variable of the parametric equation, the dependent variable representing the predictive quality value.

[0093] Example 8 includes the themes of Example 7 and further defines the following: The parametric equation defines a differentiable curve.

[0094] Example 9 includes the themes of Example 7 and further defines the following: The parametric equation defines a differentiable surface.

[0095] Example 10 includes the subject matter of any of Examples 1 to 9 and further defines the following: The different sets of reference coding parameters include randomized sets of coding parameters.

[0096] Example 11 includes the subject matter of any of Examples 1 to 10 and further defines: generating the training data by encoding each of the plurality of reference video data packages with each different set of encoding parameters.

[0097] Example 12 includes the subject matter of any of Examples 1 to 11 and further includes training the machine learning model.

[0098] Example 13 includes the subject matter of any of Examples 1 to 12 and further defines: providing metadata of the subject video data package to the machine learning model, wherein the metadata includes at least one of the file type, selected subject encoding parameters, intended playback software, and intended playback device type.

[0099] Example 14 includes the subject matter of any of Examples 1 to 13 and further specifies the following: The multiple reference video data packages are generated by the Active Encoding Platform.

[0100] Example 15 includes the subject matter of any of Examples 1 to 14 and further defines: calculating an actual quality value by comparing the encoded target video data package with the target video data package, and retraining the machine learning model using the actual quality value.

[0101] Example 16 includes the subject matter of any of Examples 1 to 15 and further defines the following: The machine learning model includes a first machine learning model configured to generate a plurality of predicted first quality values ​​and a second machine learning model configured to generate a plurality of predicted second quality values.

[0102] Example 17 includes the subject matter of any of Examples 1 to 16 and further defines the following: The neural network includes an input layer and a number of successor layers, each successor layer having a reduced width and height compared to the preceding layer, and each successor layer having a greater depth compared to the preceding layer.

[0103] Example 18 includes the subject matter of any of Examples 1 to 17 and further specifies the following: training data is repeatedly generated, and the machine learning model is repeatedly retrained using the training data.

[0104] Example 19 includes a non-temporary computer-readable medium containing executable instructions that, when executed by the machine's electronic processor, cause the machine to perform the method described in any of Examples 1 to 18.

[0105] Embodiment 20 includes a system comprising memory hardware configured to store instructions and processor hardware configured to execute the instructions. The instructions include providing a target video data package to a neural network to generate a plurality of predicted quality values ​​for the target video data package, each of the plurality of predicted quality values ​​being associated with a different set of target coding parameters from a range of coding parameters, the neural network being trained with training data comprising a plurality of reference video data packages and reference quality values ​​calculated for each reference video data package coded according to a different set of reference coding parameters from the range of coding parameters, setting the target coding parameters for the target video data package based on the plurality of predicted quality values, and sending a control signal to an encoder module to code the target video data package using the target coding parameters.

[0106] The above description is illustrative in nature and does not limit the scope or application of the Disclosure. The broad teachings of the Disclosure can be carried out in a variety of ways. While the Disclosure includes specific examples, other modifications will become apparent by examining the drawings, the description herein, and the following claims. In the description and claims, one or more steps within any given method may be performed in a different order without altering the principles of the Disclosure. Alternatively, steps may be performed simultaneously or omitted. Similarly, instructions stored in a non-temporary computer-readable medium may be performed in a different order, simultaneously, or omitted without altering the principles of the Disclosure. Unless otherwise noted, the numbering or other labeling of instructions or method steps is for convenience of reference and does not necessarily indicate a fixed order or arrangement.

[0107] Unless the context clearly indicates otherwise, the articles "a," "an," and "the" should not be interpreted as meaning "the only one." Rather, these articles should be interpreted as meaning "at least one" or "one or more." Similarly, when the terms "the" or "said" are used to refer to a noun introduced with the indefinite article "a" or "an," unless the context clearly indicates otherwise, the terms "the" or "said" should also be interpreted as meaning "at least one" or "one or more."

[0108] Spatial and functional relationships between elements (such as modules) are described using terms such as (but not limited to) "connected," "engaged," "interfaced," and / or "coupled." Unless explicitly stated as "direct," relationships between elements are either direct or may involve intervening elements. The expression "at least one of A, B, and C" should be interpreted as indicating a logical relation (A or B or C), where OR represents non-exclusive OR. It should not be interpreted as meaning "at least one of A, at least one of B, and at least one of C." The term "set" does not necessarily exclude an empty set. For example, a "set" may have zero elements. The term "subset" does not necessarily require a true subset. For example, a "subset" of set A may have the same scope as set A or may contain elements of set A. Furthermore, the term "subset" does not necessarily exclude an empty set.

[0109] In diagrams, the direction of arrows generally indicates the flow of information, such as data or instructions. However, the direction of an arrow does not necessarily mean that information cannot be transmitted in the reverse direction. For example, if information is sent from a first element to a second element, the arrow may be shown pointing from the first element to the second element. However, the second element may send a data request to the first element and / or send an acknowledgment of information receipt.

[0110] Throughout this application, the terms “module” or “controller” may be replaced with the term “circuit.” “Module” refers to, or may constitute part of, processor hardware that executes code, and memory hardware that stores the code executed by the processor hardware. The term “module” may include one or more interface circuits. In various implementations, an interfering circuit may implement a wired or wireless interface connected to, or constituting part of, a communication system. A module may communicate with other modules using an interfering circuit. In various implementations, the functionality of a module may be distributed among multiple modules connected via a communication system. For example, functionality may be distributed among multiple modules by a load balancing system. In various implementations, the functionality of a module may be divided among multiple computing platforms connected by a communication system.

[0111] The term “code” includes software, firmware, and / or microcode, and may refer to programs, routines, functions, classes, data structures, and / or data objects. The term “memory hardware” may be a subset of the term “computer-readable medium.” The term “computer-readable medium” does not include transient electrical or electromagnetic signals, or electromagnetic signals that propagate through a medium, such as on an electromagnetic carrier. The term “computer-readable medium” is considered to be tangible and non-transient. The modules, methods, and apparatus described herein may be partially or fully implemented by a dedicated computer created by configuring a general-purpose computer to perform one or more specific functions described in a computer program. The functional blocks, flowchart elements, and message sequence charts described above serve as software specifications that can be translated into a computer program through routine work by a skilled technician or programmer.

[0112] Furthermore, while certain drawings may show hardware and software located within specific devices, it should be understood that these illustrations are for illustrative purposes only. In some embodiments, the illustrated components may be integrated or separated into software, firmware, and / or hardware. For example, instead of being located within a single electronic processor and executed by it, logic circuits and processing may be distributed across multiple electronic processors. Regardless of the method of integration or separation, hardware and software components may be located on the same computing device or distributed across different computing devices (e.g., computing devices interconnected by one or more networks or other communication systems).

[0113] Where a claim states that an apparatus or system includes an electronic processor or other element having a particular configuration, the claim or the element of the claim should be interpreted as meaning one or more electronic processors (or appropriate other elements). Where a claim states that an electronic processor (or other element) is configured to make one or more decisions or to perform one or more steps, the claim should be interpreted as meaning that any combination of one or more electronic processors (or any combination of one or more other elements) may be configured to perform any combination of one or more decisions (or one or more steps).

Claims

1. A computer-based method for encoding video data based on predicted quality values ​​generated by machine learning, The method involves providing a target video data package to a neural network to generate multiple predictive quality values ​​for the target video data package, wherein each of the multiple predictive quality values ​​is associated with a different set of target coding parameters from a range of coding parameters, the predictive quality values ​​are expressed by parametric equations that define fully differentiable surfaces or curves, each representing a different type of predictive quality value, and the neural network is trained using training data containing multiple reference video data packages and reference quality values ​​calculated for each reference video data package encoded according to a different set of reference coding parameters from the range of coding parameters. Based on the aforementioned multiple predicted quality values, the target encoding parameters of the target video data package are set, Sending control signals to the encoder module to encode the target video data package using the target encoding parameters, Methods that include...

2. The aforementioned multiple predictive quality values, A first surface that defines a first prediction quality value as a function of coding parameters, and It includes a second surface that defines a second prediction quality value as a function of the coding parameters, Setting the target encoding parameters of the target video data package based on the aforementioned multiple predicted quality values ​​is: This includes selecting coding parameters that maximize the first predictive quality value while satisfying the quality threshold of the second predictive quality value, The method according to claim 1.

3. The method according to claim 1 or 2, wherein the predicted quality value includes at least one of a peak signal-to-noise ratio (PSNR) index, a structural similarity index measure (SSIM) index, a multi-scale structural similarity index measure (MS-SSIM) index, a video quality metric (VQM) index, and / or a video multi-method assessment fusion (VMAF) score, and bitrate.

4. The method according to any one of claims 1 to 3, wherein the target encoding parameter includes at least one of an encoding codec, a quantization parameter (QP), an output bitrate, and an output resolution.

5. The method according to any one of claims 1, 3, and 4, wherein the plurality of predicted quality values ​​are generated using coefficients of a predefined parametric equation.

6. The method according to claim 5, wherein the plurality of prediction quality values ​​are generated using the independent variables of the parametric equation, and the independent variables represent coding parameters.

7. The method according to claim 6, wherein the plurality of prediction quality values ​​are generated using the dependent variable of the parametric equation, and the dependent variable represents an encoding parameter.

8. The method according to claim 7, wherein the parametric equation defines a differentiable curve.

9. The method according to claim 7, wherein the parametric equation defines a differentiable surface.

10. The method according to any of the preceding claims, wherein the different sets of reference coding parameters include a randomized set of coding parameters.

11. The method according to any of the prior claims, further comprising generating the training data by encoding each of the plurality of reference video data packages with each different set of encoding parameters.

12. The method according to any of the prior claims, further comprising training the machine learning model.

13. The method according to any of the preceding claims, further comprising providing metadata of the target video data package to the machine learning model, wherein the metadata includes at least one of a file type, selected target encoding parameters, intended playback software, and intended playback device type.

14. The plurality of reference video data packages are generated by an active encoding platform, according to any of the preceding claims.

15. The method according to any of the prior claims, further comprising comparing an encoded target video data package with the target video data package to calculate an actual quality value, and retraining the machine learning model using the actual quality value.

16. The method according to any one of the preceding claims, wherein the machine learning model includes a first machine learning model configured to generate a plurality of predicted first quality values ​​and a second machine learning model configured to generate a plurality of predicted second quality values.

17. The aforementioned neural network includes an input layer and a plurality of subsequent layers. Each subsequent layer is less wide and less tall than the preceding layer. Each subsequent layer is deeper than the preceding layer. The method according to any of the prior claims.

18. Training data is repeatedly generated, The machine learning model is repeatedly retrained using the training data. The method according to any of the prior claims.

19. A non-temporary computer-readable medium comprising an executable instruction, when executed by the machine's electronic processor, causing the machine to perform the method described in any of the preceding claims.

20. Memory hardware configured to store instructions, Processor hardware configured to execute the aforementioned instructions, A system including, wherein the instruction is The method involves providing a target video data package to a neural network to generate multiple predictive quality values ​​for the target video data package, wherein each of the multiple predictive quality values ​​is associated with a different set of target coding parameters from a range of coding parameters, the predictive quality values ​​are expressed by parametric equations that define fully differentiable surfaces or curves, each representing a different type of predictive quality value, and the neural network is trained using training data containing multiple reference video data packages and reference quality values ​​calculated for each reference video data package encoded according to a different set of reference coding parameters from the range of coding parameters. Based on the aforementioned multiple predicted quality values, the target encoding parameters of the target video data package are set, Sending control signals to the encoder module to encode the target video data package using the target encoding parameters, A system that includes this.

Images