Video data encoding using machine learning techniques that predict image quality

Abstract

The present disclosure discloses a computer-implemented method for encoding video data according to predicted quality values generated by machine learning, the method comprising: providing a target video data packet to a neural network to generate a plurality of predicted quality values for the target video data packet, each of the plurality of predicted quality values is associated with a different set of target coding parameters from a range of coding parameters, the neural network being trained using training data comprising a plurality of reference video data packets and a reference quality value calculated for each reference video data packet, wherein each reference video data packet is encoded according to a different reference coding parameter set from the range of coding parameters; setting a target coding parameter of the target video data packet based on the plurality of predicted quality values; and sending a control signal to an encoder module to encode the target video data packet using the target encoding parameter.

Description

Cross-references to related applications

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 509,585, filed June 22, 2023, and European Application No. 23190618.1, filed August 9, 2023, each of which is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure relates to image analysis, and more specifically to image analysis techniques for machine learning implementations of video data. This analysis can be used to set one or more encoding parameters for the video data. Summary of the Invention

[0003] Encoding or compressing video data offers a variety of technical benefits. For example, encoded video data is often significantly smaller than the source video data. Reducing video data size, in addition to reducing storage space requirements, helps to buffer and stream video data to end users faster, reduces bandwidth requirements for content delivery systems, improves compatibility across devices and platforms, and—because smaller video packets require less computing power to process and transmit—can provide significant energy savings by reducing the computing demands on data centers and / or content delivery networks.

[0004] While encoding video data offers many benefits, lossy compression algorithms often remove or reduce certain visual information. Lossy compression algorithms can introduce artifacts and / or blurring into encoded video data—especially when using lower bitrates and / or aggressive compression settings. Furthermore, video encoding can be computationally intensive and time-consuming, and often requires users to choose from a variety of complex encoding parameters that can affect quality, file size, and compatibility. Because encoding video data can be computationally intensive and time-consuming, selecting encoding parameters through an iterative trial-and-error process is generally inefficient.

[0005] Furthermore, while video quality values ​​can be calculated to quantify the quality loss of encoded video data, these values ​​typically require comparing the encoded video data with the original source video data. Therefore, such quality values ​​can only be generated after two operations: (i) encoding and decoding the original (or source) unencoded source video data, and (ii) performing additional computational processes to compare the source video data with the encoded video data. Since the computational processes required to analyze and compare unencoded video data with encoded video data can often be almost as computationally intensive as the encoding process itself, applying video quality values ​​to estimate video quality by comparing encoded video data with source video data is generally more computationally intensive than the encoding process itself. Additionally, since quality values ​​depend on the statistics of the content (each video packet, such as a data file containing the complete content or other relevant content), it is often necessary to generate quality values ​​for each different piece of content (each different video packet) and / or regenerate quality values ​​when the encoding parameters used to encode the video data change. In some examples, each video packet may include a collection of video data associated with a single piece of video content.

[0006] Given the enormous computational demands associated with (i) encoding video data and (ii) analyzing the encoded data against the unencoded version, performing real-time or near-real-time video quality analysis on encoded video data may be computationally infeasible. Therefore, what is needed is optimized image analysis techniques that can significantly improve the computational throughput of computer systems generating encoded video with specific visual qualities. This novel and inventive high-throughput computational technique can rapidly generate encoded streams with predicted video quality and / or set encoding parameters for specific content based on the predicted video quality—in real-time or near real-time (e.g., at software modules such as video bitrate distortion controllers). This significantly reduces the overall computational burden and increases the throughput of the video data analysis and encoding process.

[0007] Various aspects of this disclosure relate to high-throughput systems and methods for analyzing source video data and generating predicted video quality values ​​within a coding parameter envelope, wherein coding parameters can be set based on such predictions. In some examples, the coding parameter envelope may include one or more ranges of one or more types of coding parameters.

[0008] For example, a computer-implemented method for encoding video data based on predicted quality values ​​generated through machine learning includes: providing a target video data packet to a neural network to generate a plurality of predicted quality values ​​for the target video data packet, each of the plurality of predicted quality values ​​being associated with a different set of target coding parameters from a range of coding parameters, the neural network being trained using training data including a plurality of reference video data packets and a reference quality value computed for each reference video data packet, wherein each reference video data packet is encoded based on a different set of reference coding parameters from the range of coding parameters; setting target coding parameters for the target video data packet based on the plurality of predicted quality values; and sending a control signal to an encoder module to encode the target video data packet using the target coding parameters.

[0009] Among other features, the plurality of predicted quality values ​​includes: a first surface that defines a first predicted quality value as a function of coding parameters; and a second surface that defines a second predicted quality value as a function of coding parameters. Setting the target coding parameters for the target video data packet based on the plurality of predicted quality values ​​includes selecting coding parameters that maximize the first predicted quality value while satisfying a quality threshold for the second predicted quality value. Among other features, the predicted quality values ​​include at least one of Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Multi-Scale Structural Similarity Index (MS-SSIM), Video Quality Metric (VQM), Video Multi-Method Evaluation Fusion (VMAF) score, and bitrate.

[0010] In other features, the target coding parameters include at least one of a codec, quantization parameter (QP), output bitrate, and output resolution. In other features, the plurality of predicted quality values ​​are represented as coefficients of a predefined parametric equation. In other features, the plurality of predicted quality values ​​include independent variables of the parametric equation, and the independent variables represent coding parameters. In other features, the plurality of predicted quality values ​​include dependent variables of the parametric equation, and the dependent variables represent predicted quality values. In other features, the parametric equation defines a differentiable curve. In other features, the parametric equation defines a differentiable surface. In other features, the method includes generating the training data by encoding each of the plurality of reference video data packets using each different set of coding parameters.

[0011] Among other features, the method includes: providing the machine learning model with metadata of the target video data packet, the metadata including at least one of file type, selected target encoding parameters, expected playback software, and expected playback device type. Among other features, the method includes: calculating an actual quality value by comparing an encoded target video data packet with the target video data packet, and retraining the machine learning model using the actual quality value. Among other features, the machine learning model includes a first machine learning model configured to generate a plurality of predicted first quality values ​​and a second machine learning model configured to generate a plurality of predicted second quality values. Among other features, a non-transitory computer-readable medium includes executable instructions that, when executed by an electronic processor of a machine, cause the machine to perform the method.

[0012] In some embodiments, a system includes: memory hardware configured to store instructions; and processor hardware configured to execute the instructions. The instructions include: providing a target video data packet to a neural network to generate a plurality of predicted quality values ​​for the target video data packet, each of the plurality of predicted quality values ​​being associated with a different set of target coding parameters from a range of coding parameters, the neural network being trained using training data including a plurality of reference video data packets and a reference quality value computed for each reference video data packet, wherein each reference video data packet is a computed reference quality value encoded based on a different set of reference coding parameters from the range of coding parameters; setting target coding parameters for the target video data packet based on the plurality of predicted quality values; and sending a control signal to an encoder module to encode the target video data packet using the target coding parameters.

[0013] Other examples, embodiments, features, and aspects will become apparent from consideration of the specific implementation and the accompanying drawings. Attached Figure Description

[0014] Figure 1 This is a block diagram of an example implementation of a system for implementing machine learning enabling technologies that analyze source video packets and generate predicted video quality values ​​within an encoding parameter envelope.

[0015] Figure 2 This is a flowchart illustrating an example process for training a machine learning model and using the trained machine learning model to analyze source video data packets and generate predicted video quality for any given set of encoding parameters or within the entire encoding parameter envelope.

[0016] Figures 3A to 3B This is a flowchart of an example process for generating training data suitable for training machine learning models.

[0017] Figure 4 This is a flowchart of an example process for calculating objective quality values ​​for encoded video data.

[0018] Figure 5 This is a flowchart of an example process for calculating the structural similarity index, a quality metric, of encoded video data.

[0019] Figure 6 This is a flowchart illustrating an example process for converting video data into video tensors suitable for input into a neural network.

[0020] Figures 7A to 7B This is a flowchart of an example process for training a machine learning model.

[0021] Figure 8 and Figure 9 Example predicted quality values ​​generated by a trained machine learning model are shown.

[0022] In the accompanying drawings, reference numerals may be used repeatedly to identify similar and / or identical elements. Detailed Implementation

[0023] Figure 1 This is a block diagram of an example implementation of system 100, which includes a machine learning platform 102 and a video processing platform 104. As will be described herein, system 100 analyzes source video data (such as unencoded video data and / or video data that has been encoded at extremely high bitrates and / or quality settings [e.g., shallowly compressed video files encoded using the JPEG 2000 standard]) and generates predicted image quality values ​​within an encoding parameter envelope. For example, the encoding parameter envelope may include one or more ranges of one or more types of encoding parameters. The predicted image quality values ​​are used to set the encoding parameters of the source video data—e.g., at a rate-distortion controller. Figure 1As shown, system 100 may include communication system 106. Platforms of system 100—such as machine learning platform 102 and video processing platform 104—may communicate via communication system 106. Examples of communication system 106 may include one or more networks, such as General Packet Radio Service (GPRS) networks, Time Division Multiple Access (TDMA) networks, Code Division Multiple Access (CDMA) networks, Global System for Mobile Communications (GSM) networks, Enhanced Data Rate GSM Evolution (EDGE) networks, High Speed ​​Packet Access (HSPA) networks, Evolved High Speed ​​Packet Access (HSPA+) networks, Long Term Evolution (LTE) networks, Global Microwave Interconnection Access (WiMAX), 5G mobile networks, Internet Protocol (IP) networks, Wireless Application Protocol (WAP) networks, or IEEE 802.11 standard networks, and any suitable combination of the above networks. In various embodiments, communication system 106 may also include optical networks, local area networks, and / or global communication networks, such as the Internet.

[0024] In some examples, the machine learning platform 102 includes a shared system resource 108, a communication interface 110, and one or more data storage devices, such as storage device 112, including a non-transitory computer-readable storage medium. In some embodiments, the shared system resource 108 includes one or more electronic processors, one or more graphics processing units, volatile computer memory, non-volatile computer memory, and / or one or more system buses connecting the shared system resource 108, the communication interface 110, and / or the storage device 112. In various embodiments, the storage device 112 includes one or more software modules, such as a machine learning training module 120 and / or a machine learning module 122. Additional functionality of the machine learning training module 120 and the machine learning module 122 will be referenced in this specification. Figures 2 to 7B Further description.

[0025] In some embodiments, the video processing platform 104 includes a shared system resource 114, a communication interface 116, and one or more data storage devices, such as storage device 118, including a non-transitory computer-readable storage medium. In some examples, the shared system resource 114 includes one or more electronic processors, one or more graphics processing units, volatile computer memory, non-volatile computer memory, and / or one or more system buses connecting the shared system resource 114, the communication interface 116, and / or the storage device 118. In various embodiments, the storage device 118 includes one or more software modules, such as a video encoding module 124, a video analysis module 126, and / or a rate-distortion controller module 128. Additional functionality of the video encoding module 124, the video analysis module 126, and / or the rate-distortion controller module 128 will be referenced in this specification. Figures 2 to 7BFurther description.

[0026] In various implementations, components of the machine learning platform 102 communicate with components of the video processing platform 104 via the communication system 106. For example, components of the machine learning platform 102 communicate with communication interface 110, and components of the video processing platform 104 communicate with communication interface 116. Communication interfaces 110 and 116 can then communicate with each other via the communication system 106.

[0027] Although machine learning platform 102 and video processing platform 104 are in Figure 1 While shown as two separate platforms, machine learning platform 102 and video processing platform 104 can also be integrated into the same platform. For example, shared system resources 108 and 114 may include the same components. Similarly, storage device 112 and storage device 114 may include the same components. In some embodiments, machine learning platform 102 and video processing platform 104 can communicate via dedicated wired and / or wireless connections. In some examples, machine learning platform 102 may be implemented as a sub-component of video processing platform 104.

[0028] Figure 2 This is a flowchart of an example process 200 for (i) training a machine learning model, (ii) analyzing source video data using the trained machine learning model, and (iii) generating predicted image quality values ​​for any given set of coding parameters or for the entire coding parameter envelope. It should be understood that the training process described herein is optional, as this part can be performed by a separate process, wherein, in such an embodiment, process 200 only includes the analysis and value generation as described herein. Process 200 is described herein as being performed by machine learning platform 102. However, it should be understood that process 200, or a portion thereof, can be performed by a collection of platforms or devices.

[0029] At 202, the machine learning training module 120 generates training data. The machine learning training module 120 can generate training data from a reference video database—such as reference video data stored in storage devices 112 and / or 118. In some examples, the training data includes source video data as reference video data. For each reference video packet (such as video data associated with each piece of content), the training data may also include: (i) one or more different sets of encoding parameters; (ii) reference quality values ​​computed for the reference video packet after encoding according to each of the different sets of encoding parameters; and / or (iii) one or more reference video tensors suitable for input into the machine learning model. In some embodiments, the encoding parameters include parameters of different categories. For example, encoding parameters may include a codec, quantization parameters (QP) of the encoder, output bitrate, output resolution, output frame rate, output audio parameters, constant bitrate factor (CRF) configuration, group of pictures (GOP) length, and / or buffer size. In some examples, the one or more encoding parameters include a combination of parameters of a single category. In other examples, the one or more encoding parameters include a combination of parameters of multiple categories.

[0030] For example, different sets of coding parameters can include different combinations of first coding parameters (such as QP values) and second coding parameters (such as code rate values) within the coding parameter envelope. As a non-limiting example, if the coding parameters include... n A unique first encoded parameter and m If there is a unique second encoding parameter, then different sets of encoding parameters can include the first encoding parameter and the second encoding parameter. A unique combination. In some examples, the unique combination of the first and second encoded parameters is stored in... In an array or matrix. In some examples, the first encoding parameter includes QP values ​​in the range of about 16 to about 32. In other examples, the first encoding parameter includes QP values ​​in the range of about 0 to about 20. In various implementations, the first encoding parameter includes QP values ​​in the range of about 20 to about 30. In some embodiments, the first encoding parameter includes QP values ​​in the range of about 30 to about 40. In an example embodiment, the first encoding parameter includes QP values ​​in the range of about 40 to about 51.

[0031] In various implementations, the second encoding parameter includes an output resolution ranging from approximately 320 vertical lines to approximately 3840 vertical lines. In some examples, the second encoding parameter includes an output bitrate value ranging from approximately 15 megabits per second (Mbps) to approximately 40 Mbps. In example embodiments, the second encoding parameter includes an output bitrate value ranging from approximately 8 Mbps to approximately 12 Mbps. In other examples, the second encoding parameter includes an output bitrate value ranging from approximately 2 Mbps to approximately 8 Mbps. In various implementations, the second encoding parameter includes an output bitrate value ranging from approximately 300 kilobits per second (Kbps) to approximately 2 Mbps. In some embodiments, the encoder codec may be specified as a High Efficiency Video Coding (HEVC) Main10 codec. For example, the encoder codec may be an X.264 codec for creating digital video streams in H.264 / MPEG-4 AVC video encoding format. In other examples, the encoder codec may be an X.265 codec for creating digital video streams in HEVC / H.265 video compression format.

[0032] Reference quality values ​​can include values ​​used to evaluate the performance and effectiveness of video codecs and encoding settings—such as, for example, quantifying how well the encoded video data maintains the visual quality of the original source video data while minimizing artifacts and maintaining an acceptable file size or bitrate. Examples of quality values ​​include objective metrics such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM) metric, Multi-Scale Structural Similarity Index (MS-SSIM) metric, Video Quality Metric (VQM) metric, and / or Video Multi-Method Evaluation Fusion (VMAF) score. In various implementations, one or more reference quality values ​​may be used for each reference video packet. For each video packet, the reference quality value may include a reference quality value corresponding to each unique combination of encoding parameters (each distinct set of encoding parameters). For example, a reference quality value for a given combination of encoding parameters can indicate how a reference video packet encoded using a particular combination of encoding parameters performs relative to a corresponding source video packet. In various implementations, the reference video data may be preprocessed and transformed into a format suitable for input into machine learning models—such as neural networks. For example, each reference video packet may be preprocessed and transformed into a corresponding reference video vector, array, and / or (one or more) tensor. Additional details related to generating training data at position 202 will be referenced. Figures 3A to 3B Describe it.

[0033] At 204, the machine learning training module 120 uses training data—such as the training data generated at 202—to train a machine learning model. In some examples, the machine learning model includes a neural network—such as a convolutional neural network (CNN). Examples of suitable neural networks include 3D convolutional neural networks (3D-CNN), convolutional long short-term memory recurrent neural networks (ConvLSTM), two-stream convolutional networks, long-term recurrent convolutional networks (LRCN), networks based on temporal distribution layers, and / or transformer-based networks. In various implementations, the trained machine learning model may be stored in storage device 112 and / or storage device 118. Additional details associated with training a machine learning model using training data will be referenced in [reference]. Figures 7A to 7B Describe it.

[0034] At 206, machine learning module 122 provides (i) a target video data packet and optionally (ii) one or more different sets of encoding parameters (such as any of the encoding parameters described previously) to a trained machine learning model to generate a predicted quality value for each different set of encoding parameters. The target video data packet and optional encoding parameters can be entered or selected by a user on a graphical user interface—for example, through graphical user interface elements that generate and output to the display of a separate user device (such as a workstation, terminal, mobile phone, tablet computer, or any other computing device) connected to machine learning platform 102 and / or video processing platform 104. In various embodiments, the user selects a user interface element corresponding to the target video data packet, and—in response to the user's selection—machine learning platform 102 retrieves the target video data packet from one or more data storage devices (such as storage device 112 and / or storage device 118).

[0035] In some embodiments, the generated predicted quality values ​​can be output to a graphical user interface. In various implementations, the encoding parameters can include any category of encoding parameters included in the training data used to train the machine learning model. In some examples, the trained machine learning model can generate predicted quality values ​​for each combination of encoding parameters (each different set of encoding parameters). In example embodiments, the trained machine learning model can generate predicted quality values ​​that indicate the predicted quality assessment of a target video data packet if encoded using a specific combination (set) of encoding parameters. For example, if the encoding parameters provided at 206 include... n The first encoding parameter and m Then, the machine learning model can generate a second encoding parameter. A predicted quality value—such as one or more predicted quality values ​​associated with each combination of coding parameters.

[0036] In various implementations, the predicted quality value can be expressed as: Arrays. For example, predicted values ​​from various quality values ​​can be stored as additional dimensions in each combination of encoding parameters. In some embodiments, this can be a combination of random sampling coding parameters. Generate predicted quality values. In other examples, predicted quality values ​​can be expressed in parametric form—for example, as an equation defining a differentiable curve or differentiable surface. If the predicted quality values ​​comprise predicted quality values ​​for a single class across the entire range of encoded parameters for a single class, then the predicted quality values ​​can be expressed as an equation defining a differentiable curve. If the predicted quality values ​​comprise predicted quality values ​​for a single class across the entire range of encoded parameters for two different classes, then the predicted quality values ​​can be expressed as an equation defining a differentiable surface. In various examples, the trained machine learning model can output one or more parametric equations. In some implementations, the trained machine learning model can output the coefficients of one or more parametric equations.

[0037] Equation (1) below illustrates the effect for any given input coding parameters. Output predicted quality value function :

[0038] In equation (1) above, Differentiable curves can be defined, and functions This represents a predefined function. Coefficients and Generated by a trained machine learning model. Therefore, the output coefficients of the trained machine learning model... and Afterwards, the user can add any encoding parameters. Input it into equation (1) and quickly calculate the given encoding parameters. Predicted quality value .

[0039] Equation (2) below illustrates the effect for any given combination of input coding parameters. Output predicted quality value function :

[0040] In equation (2) above, Differentiable surfaces can be defined, and functions This represents a predefined function. Coefficients and Generated by a trained machine learning model. Similar to equation (1), the coefficients are generated by the trained machine learning model output. and Afterwards, users can combine any encoding parameters. Input it into equation (1) and quickly compute the given combination of coding parameters. Predicted quality value In some examples, if the user already knows the expected predicted quality value... and / or Then equations (1) and / or (2) can be solved to obtain the coding parameters. In various implementations, the concepts described in reference equations (1) and (2) above can be scaled to include any number and combination of input encoding parameters and any number and combination of output prediction quality values. For example, a machine learning model can be designed for any given combination of input encoding parameters. Output two predicted quality values and .

[0041] At 208, the machine learning module 122 and / or the rate-distortion controller module 128 select one or more of a plurality of predicted quality values ​​based on one or more threshold quality values. For example, the machine learning module 122 and / or the rate-distortion controller module 128 may load one or more threshold quality values. The threshold quality value may represent a minimum or target value for a particular quality value. For example, the threshold quality value may include a target value for one or more visual quality values ​​(VMAF, SSIM, etc.). In some embodiments, a user may input the threshold quality value into an input field in a graphical user interface. The machine learning module 122 and / or the rate-distortion controller module 128 may parse the predicted quality values ​​to select one or more of the predicted quality values ​​for the target video. For example, the machine learning module 122 and / or the rate-distortion controller module 128 may compare the predicted quality values ​​output from a trained machine learning model with one or more threshold quality values ​​to identify one or more predicted quality values ​​that are closest to the target quality value.

[0042] The machine learning model is configured to output two predicted quality values. and In the example, machine learning module 122 and / or rate distortion controller module 128 can be configured to find ways to improve quality values. and Maximize encoding parameters For example, machine learning module 122 and / or rate distortion controller module 128 are... Set a first threshold and find encoding parameter combinations that meet or exceed the first threshold. The machine learning module 122 and / or rate distortion controller module 128 are... Set a second threshold and look for combinations of encoding parameters that meet or exceed the second threshold. In some embodiments, the machine learning module 122 and / or the rate distortion controller module 128 search for conditions that meet or exceed the specified values. The first quality threshold and The second quality threshold is a combination of the two encoding parameters. For example, machine learning module 122 and / or rate distortion controller module 128 search for entities that exist in... and Combination of encoding parameters in both .

[0043] In various implementations, the machine learning module 122 and / or the rate distortion controller module 128 selects to make the first quality value Maximize while still satisfying the second quality value Combination of encoding parameters for quality constraints (e.g., a second quality threshold). For example, machine learning module 122 and / or rate distortion controller module 128 from Select to make Maximize and also located by The described combination of encoded parameters on the curved surface. In some embodiments, the machine learning module 122 and / or the rate-distortion controller module 128 select to make the second quality value... Maximize while still satisfying the first quality value Combination of encoding parameters for quality constraints (e.g., a first quality threshold) For example, machine learning module 122 and / or rate distortion controller module 128 from Select to make Maximize and also located by The combination of encoded parameters on the described surface.

[0044] At 210, the machine learning module 122 and / or the rate-distortion controller module 128 set target coding parameters for the target video based on a set of coding parameters associated with a selected predicted quality value from a plurality of predicted quality values. For example, the machine learning module 122 and / or the rate-distortion controller module 128 set target coding parameters for the target video based on the set of coding parameters selected at 210. Target coding parameters may include coding parameters associated with the selected predicted quality values, or may be generated based on these parameters. For example, in some embodiments, the coding parameters associated with the selected values ​​may be modified (e.g., scaled) when setting the target coding parameters for the target video, depending on how close the corresponding value is to a threshold. As described above, the predicted quality values ​​may be parametric equations, and in this implementation, the machine learning module 122 and / or the rate-distortion controller module 128 solves the parametric equations to determine the target coding values ​​required to achieve the desired target quality value.

[0045] In various implementations, contextual data (such as metadata) may be provided to the machine learning module 122 and / or the rate-distortion controller module 128 (e.g., provided along with the target video data at 206). Examples of metadata include at least one of file type, selected target encoding parameters, expected playback software, and expected playback device type. The machine learning module 122 may automatically select target encoding parameters from predicted quality values ​​based on the metadata.

[0046] The target encoding parameters can be output to a graphical user interface. At 212, the machine learning module 122 and / or the rate-distortion controller module 128 pass the target encoding parameters to the video encoding module 124, and the video encoding module 124 uses the target encoding parameters to encode the target video into encoded video. In various embodiments, the encoded video is stored in storage device 112 and / or storage device 118. In some examples, the machine learning module 122 and / or the video encoding module 124 calculates an actual quality value by comparing the encoded target video data packet with the target video data packet. The machine learning module 122 then uses the actual quality value to retrain the machine learning model.

[0047] In some embodiments, the first predicted quality value generated at 206 The prediction quality can be represented by the following equation (7). :

[0048] In equation (7) above, Defined as a variable and The surface of a function. For example, These can be the first encoder parameters (such as quantization parameters). This can be a second encoder parameter (such as resolution), and , , and These can be coefficients predicted by a machine learning model for a given input image.

[0049] In various implementations, the second predicted quality value is generated at position 206. The predicted code rate can be represented by the following equation (8). :

[0050] As in equation (7), Defined as a variable and The surface of the function, and , , and These can be coefficients predicted by a machine learning model for a given input image. In some examples, at 208, the rate-distortion controller module 128 indicates the target bitrate. Maximize quality To select the target quality value. For example, since a target bitrate is defined. The value of is obtained by solving equation (8). and One of them. In various implementations, the rate-distortion controller module 128 first determines based on Solve For example, equation (8) can be expressed as the following quadratic equation (9), and solved using the quadratic formula to obtain... .

[0051]

[0052] Rate-distortion controller module 128 then solves equation (9) ( Substituting the solution into equation (7) (mass) In the curved surface, this results in two different equations. Each equation is a single variable. The function. Rate-distortion controller module 128 then searches for these two different equations. The maximum value of each equation in the equation. For example, the rate distortion controller module 128 calculates these two different equations. The derivative of each equation in Set the derivative to zero and solve. Rate-distortion controller module 128 then through Substitute into the solution of equation (9) to find The corresponding value. Rate-distortion controller module 128 then uses and All valid combinations are used to test equation (7). Rate-distortion controller module 128 will be designed for... and Each combination calculated The value is saved to an array .

[0053] At position 210, rate-distortion controller module 128 from... Select with The highest calculated value corresponding to and The combination. At 212, the rate-distortion controller module 128 uses the selected... (e.g., quantization parameters) and The target video data packets are encoded using a combination of factors (e.g., resolution). In various implementations, the rate-distortion controller module 128 will select... and The combination is passed to the video encoding module 124, and the video encoding module 124 uses the selected and The combination of these methods encodes the target video data packets.

[0054] Although Figure 2 This example illustrates a single machine learning model generating multiple types of predicted quality values; however, in other implementations, multiple machine learning models can also be used to generate predicted quality values. For instance, each of multiple machine learning models can be used to generate a specific type of predicted quality value. Therefore, in an example requiring two types of predicted quality values, a first machine learning model can generate a first type of predicted quality value, and a second machine learning model can generate a second type of predicted quality value. In various implementations, a single machine learning model can be trained to output more than one type of predicted quality value. For example, a single machine learning model can be trained to output both a first type of predicted quality value and a second type of predicted quality value. In some examples, a single machine learning model can be trained to output any number of types of predicted quality values.

[0055] Figures 3A to 3B This is a flowchart of an example process 300 for generating training data. At 302, the machine learning training module 120 can generate combinations of encoded parameters. For example, the machine learning training module 120 selects encoded parameters for one or more categories—such as those previously referenced. Figure 2The machine learning training module 120 describes the encoding parameters and generates random combinations of encoding parameters for each category. At 304, the machine learning training module 120 instructs the video encoding module 124 to load multiple reference video data packets. These reference video data packets can be source video data packets selected for training the machine learning model. At 306, the machine learning training module 120 instructs the video encoding module 124 to select an initial reference video data packet from the multiple reference video data packets. At 308, the machine learning training module 120 instructs the video encoding module 124 to select an initial combination of encoding parameters from the encoding parameters generated at 302. At 310, the machine learning training module 120 instructs the video encoding module 124 to encode the selected reference video data packet using the selected combination of encoding parameters. The encoded reference video data packet can be saved to storage device 112 and / or storage device 118.

[0056] At 312, the machine learning training module 120 instructs the video analysis module 126 to generate a reference quality value for the encoded video data packet. For example, the encoded video data packet is first decoded. During decoding, the encoded video data packet is interpreted according to the codec used to encode the video data packet, and the video data packet is reconstructed. Decoding may be important because certain effects during the encoding process—such as loss of detail, blurring, blockiness, and / or ringing—may only become apparent after the video data packet is reconstructed. After the encoded video data packet is decoded, the machine learning training module 120 generates an objective reference quality value by comparing the decoded video data packet with a reference video data packet. Additional details associated with generating the reference quality value will be referenced in this specification. Figures 4 to 6 Further description. At 314, the machine learning training module 120 associates the generated reference quality value with the selected reference video data packet and the selected combination of encoding parameters. The selected reference video data packet, the generated reference quality value, and / or the selected combination of encoding parameters can be stored as training data.

[0057] At 316, the machine learning training module 120 determines whether there exists another combination of encoding parameters that has not yet been processed at 310 to 314. In response to determining that another unprocessed combination of encoding parameters exists ("Yes" at decision box 316), the machine learning training module 120 instructs the video encoding module 124 to select the next combination of encoding parameters and encode the selected video data packet at 310 using the selected next combination of encoding parameters. In response to determining that no other unprocessed combination of encoding parameters exists ("No" at decision box 316), the machine learning training module 120 may compile the generated training data and perform any processing required to input the training data into the machine learning model. For example, the machine learning training module 132 may convert the selected reference video data packet into one or more reference video tensors suitable for input into the machine learning model. The one or more reference video tensors are associated with the selected reference video data packet in the training data. Additional details related to converting the reference video data packet into a video tensor will be further described in this specification with reference to FIG7.

[0058] At 322, the machine learning training module 120 determines whether there is another reference video packet among the plurality of reference video packets that has not yet been processed at 308 to 320. In response to determining that another unprocessed reference video packet exists ("Yes" at decision box 322), the machine learning training module 120 instructs the video encoding module 124 to select the next reference video packet at 324 and to select the initial encoding parameter combination again at 308. In response to determining that no other unprocessed reference video packet exists ("No" at decision box 322), the machine learning training module 120 saves the training data. In various embodiments, the training data may be saved to storage device 112 and / or storage device 118.

[0059] In response to determining that another set of unprocessed quality values ​​exists ("Yes" at decision box 336), the machine learning training module 120 selects the next set of reference quality values ​​at 338 and again determines at decision box 332 whether the selected reference quality value exceeds or equals the target quality value. In response to determining that no other set of unprocessed reference quality values ​​exists ("No" at decision box 336), the machine learning training module 120 converts the selected reference video data packet into one or more reference video tensors at 320. Additional details related to converting video data packets into one or more video tensors will be referenced in this specification. Figure 6 Further description.

[0060] Figure 4This is a flowchart of an example process 400 for calculating objective quality values—such as Video Multi-Method Evaluation Fusion (VMAF) scores—of encoded video data (e.g., after the encoded video data has been decoded). While process 400 describes the generation of a specific type of quality value, it should be understood that the quality values ​​are not limited to perceptual quality values ​​(such as VMAF scores). Instead, any suitable quality value (such as any combination of quality values ​​described previously with reference to process 200) can be used. At 402, video analysis module 126 loads the encoded video data packet. At 404, video analysis module 126 loads a reference video data packet corresponding to the encoded video data packet. For example, video analysis module 126 loads the source video data packet used to generate the encoded video data packet as the reference video data packet. At 406, video analysis module 126 prepares and aligns the encoded video data packet and the reference video data packet. For example, video analysis module 126 can prepare the video data packet by analyzing and / or processing the video data packet to ensure that the encoded video data packet and the reference video data packet have the same resolution, frame rate, and / or duration. Video analysis module 126 can also align the encoded video data packet and the reference video data packet to ensure that they are synchronized. At 408, the video analysis module 126 decodes the aligned reference video data packets into a first frame sequence. At 410, the video analysis module 126 decodes the aligned encoded video data packets into a second frame sequence.

[0061] At 412, the video analysis module 126 calculates the principal value for each pair of corresponding frames from the first and second frame sequences. In various embodiments, the principal value may include a mean squared error (MSE) metric, a SSIM metric, a visual information fidelity (VIF) metric, a loss of detail (DLM) metric, and / or a mean in-place pixel difference (MCPD) metric. At 414, the video analysis module 126 normalizes the principal value to a common scale. For example, each principal metric may be normalized to a scale from 0 to 100. At 416, the video analysis module 126 provides the normalized principal value to a trained machine learning model to generate a perceptual quality metric for each pair of corresponding frames. In various embodiments, the trained machine learning model may be a VMAF model, and the perceptual quality metric may be a VMAF score. At 418, the video analysis module 126 generates an average perceptual quality metric for the encoded video data packet by averaging the perceptual quality values ​​for each pair of frames in the video data packet.

[0062] Figure 5This is a flowchart of an example process 500 for calculating the structural similarity index quality metric for encoded video data packets. While process 500 describes generating a specific type of quality value, it should be understood that the quality value is not limited to the structural similarity index metric. Instead, any suitable quality value (such as any combination of quality values ​​described previously in reference process 200) can be used. At 502, the video analysis module 126 loads the encoded video data packets. At 504, the video analysis module 126 loads a reference video data packet corresponding to the encoded video data packets. For example, the reference video data packet could be an unencoded source video data packet used to generate the encoded video data packets. At 506, the video analysis module 126 aligns the reference video data packets and the encoded video data packets to ensure that these video data packets are synchronized. At 508, the video analysis module 126 extracts corresponding frames from the reference video data packets and the encoded video data packets. At 510, the video analysis module 126 converts the extracted frames to grayscale. At 512, the video analysis module 126 calculates the structural similarity index metric value for each pair of corresponding converted frames. In various embodiments, the following equation (3) can be used for each reference frame. and encoded frames right Calculate the structural similarity index measure .

[0063] (3) As shown in equation (3) above, each reference frame and encoded frames right Structural similarity index measure It can be a frame brightness comparison function Frame contrast comparison function Frame structure comparison function The product of . In various implementations, any given reference frame can be calculated according to the following equation (4). and encoded frames right Brightness comparison function .

[0064] (4) As shown in equation (4), the brightness comparison function It can be calculated as the average pixel intensity of the reference frame. Average pixel intensity of encoded frames The product of the two is divided by the sum of the numerator and denominator. In some embodiments, when the denominator is close to zero, a constant can be added to both the numerator and denominator. To avoid instability.

[0065] In some examples, any given reference frame can be calculated according to the following equation (5). and encoded frames right Contrast comparison function .

[0066] (5) As shown in equation (5), the contrast comparison function The standard deviation of pixel intensity that can be calculated as the reference frame Standard deviation of pixel intensity from coded frame The product of the two is divided by the sum of the numerator and denominator. In some embodiments, when the denominator is close to zero, a constant can be added to both the numerator and denominator. To avoid instability.

[0067] In some examples, any given reference frame can be calculated according to the following equation (6). and encoded frames right Structural comparison function .

[0068] (6) As shown in equation (6), the contrast comparison function It can be calculated as a reference frame. and encoded frames covariance of pixel intensity Divide by the average pixel intensity of the reference frame Average pixel intensity of encoded frames The product of the numerator and denominator. In some embodiments, when the denominator is close to zero, a constant can be added to both the numerator and denominator. To avoid instability.

[0069] At position 514, the video analysis module 126 generates the average structural similarity index value of the encoded video data packet by averaging the structural similarity index values ​​of each frame pair of the video data packet.

[0070] Figure 6This is a flowchart of an example process 600 for generating video tensors suitable for input into a neural network. At 602, machine learning training module 120, machine learning module 122, video encoding module 124, and / or video analysis module 126 load video data. At 604, machine learning training module 120, machine learning module 122, video encoding module 124, and / or video analysis module 126 extract frames from the loaded video data. At 606, machine learning training module 120, machine learning module 122, video encoding module 124, and / or video analysis module 126 rescale the extracted frames to the desired size required by the neural network. At 608, machine learning training module 120, machine learning module 122, video encoding module 124, and / or video analysis module 126 normalize the pixel values ​​in each extracted frame. In some examples, the pixel values ​​may be normalized to a range between approximately 0 and approximately 1. In various implementations, the pixel values ​​may be normalized to a range between approximately -1 and approximately 1.

[0071] At 610, machine learning training module 120, machine learning module 122, video encoding module 124, and / or video analysis module 126 convert the processed frames into multidimensional input matrices (or tensors). In various implementations, the RGB channels of each frame can be separated. This results in each frame having three two-dimensional matrices. In some embodiments, the processed frames can be stacked along the time dimension to create a three-dimensional matrix. or higher dimensional matrix Therefore, a video tensor is a spatiotemporal representation of video data.

[0072] Figures 7A to 7BThis is a flowchart of an example process 700 for training a machine learning model. At 704, the machine learning training module 120 initializes the machine learning model. In some examples, the machine learning model may include a neural network. The machine learning training module 120 may define the architecture of the neural network—including the number of layers, the number of neurons per layer, and the activation function. The machine learning training module 120 may initialize the weights and biases of the neural network using small random values—for example, using Xavier or He initialization methods. In various implementations, the size of the input layer is set to accept a video tensor of a single-channel (luminance or Y) 4K (3840 × 2160 pixels) video. For example, every three adjacent frames of 4K video may be stitched together to produce an input layer of size 3840 × 2160 × 3. In some embodiments, subsequent layers of the neural network are constructed using learnable convolutional filters that gradually decrease in width and height but gradually increase in depth. In an example implementation, the size of the output layer is set to correspond to an index in a defined table of possible encoding parameters. In other examples, the size of the output layer is set to directly return the optimal encoding parameters (e.g., scalar values ​​corresponding to QP and resolution).

[0073] At 704, the machine learning training module 120 loads training data—such as the training data generated at 202 of process 200 and / or according to process 300. At 706, the machine learning training module 120 divides the training data into batches. In various implementations, a batch includes a predefined number of reference videos (and their corresponding reference quality values, encoding parameters, and / or (one or more) video tensors). In some examples, a batch includes 32, 64, or 128 reference videos. In various implementations, each video is divided into multiple batches, and the size of each batch is set to include a certain length of video. For example, the size of each batch may be set to include tensors associated with a two-second video clip.

[0074] At 708, the machine learning training module 120 selects an initial batch of training data. At 710, the machine learning training module 120 selects an initial reference video data packet from the selected batch. At 712, the machine learning training module 120 provides the reference video tensor and / or encoding parameters associated with the selected reference video data packet to the initialized neural network to generate an output. At 714, the machine learning training module 120 calculates the error between this output and a reference quality value associated with the selected reference video data packet. At 716, the machine learning training module 120 determines whether the end of the selected batch has been reached. In response to determining that the end of the selected batch has not been reached ("No" at decision box 716), the machine learning training module 120 selects the next reference video data packet from the selected batch at 718, and provides the reference video tensor and / or encoding parameters associated with the selected reference video data packet to the initialized neural network at 712. In response to determining that the end of the selected batch has been reached ("Yes" at decision box 716), the machine learning training module 120 calculates the average loss of the selected batch at 720 by averaging the errors of all reference video data packets in the selected batch (e.g., the error calculated at 714).

[0075] At 724, the machine learning training module 120 uses the chain rule of calculus to calculate the gradient of the average loss relative to the weights and biases of the neural network. At 726, the machine learning training module 120 uses the calculated gradient and a predefined learning rate to update the weights and biases of the neural network. At 728, the machine learning training module 120 determines whether the end of an epoch has been reached. In some examples, an epoch is completed after calculating the error of the data associated with each reference video packet in the training dataset. In other words, an epoch is completed after the machine learning training module 120 iterates through the entire training dataset. In response to determining that the end of an epoch has not yet been reached ("No" at decision box 728), the machine learning training module 120 selects the next batch at 730 and selects the initial reference video packet from the selected batch at 710.

[0076] In response to determining that the end of the epoch has been reached ("Yes" at decision box 728), the machine learning training module 120 determines whether the training condition is met. In some examples, the training condition may be whether the average error between the output and the reference quality value is below a threshold. In various implementations, the threshold may be about 5%. In other examples, the training condition may be met after a certain number of epochs. In response to determining that the training condition has not yet been met ("No" at decision box 732), the machine learning training module 120 selects an initial batch again at 708. In response to determining that the training condition has been met ("Yes" at decision box 732), the machine learning training module 120 saves the neural network with updated weights and biases as the trained neural network.

[0077] Figure 8 and Figure 9 Example predicted quality values ​​generated by a trained machine learning model are shown. Figure 8 and Figure 9 In the example, the predicted quality value is a parametric equation that defines the surface as a function of resolution and bit rate, respectively.

[0078] The systems and methods described in this specification offer a variety of novel and inventive solutions to technical problems related to digital video analytics systems. For example, as previously mentioned, source video packets—especially high-resolution, high-bitrate packets—often tend to be large. To reduce the computational, storage, and / or energy requirements of data centers and / or content distribution systems, it is generally desirable to encode video packets to reduce their file size. Because encoded video packets are typically smaller than the original source video packets, they often require less storage space (and are often computationally more efficient when distributing and playing them). However, as previously discussed, finding optimal encoding parameters can be difficult and computationally intensive. While quality values—such as those described in this specification—quantitatively describe the relative quality loss of encoded video data compared to its unencoded source, the cumulative process of generating encoded video data objects and calculating their quality values ​​is ultimately more computationally intensive than initially encoding the video data objects. For example, generating quality values ​​requires the user to: (i) first encode the source video data object, and then (ii) perform a series of computationally intensive operations that compare the encoded video data object with the unencoded source video data object to generate the quality value. Performing this operation in real-time or near real-time is generally not feasible.

[0079] The systems and methods described in this specification provide a novel and inventive solution that enables the generation of predictive performance values ​​(or the entire predictive performance value envelope for the target video data and a given combination of coding parameters) without first encoding the target video data and performing computationally intensive comparison operations. By significantly reducing the computational operations required for users to access predictive performance values, the computational throughput of any computer system that computes these performance values ​​can be greatly improved.

[0080] Furthermore, the systems and methods described in this specification enable the generation of the entire range of predicted quality values ​​for the entire envelope of encoded parameters at once, rather than generating a single set of quality values ​​corresponding to specific encoded parameters used to encode video data. In the example of generating the range of predicted quality values ​​in parametric form, quality values ​​for combinations of encoded parameters (even those not input into and / or used to train machine learning models) can be extracted with high precision (and in real-time or near real-time). Additionally, after the range of quality values ​​has been generated (e.g., as independent numerical values ​​or in parametric form), the encoded parameters required to achieve any desired quality value can be determined quickly and efficiently in real-time or near real-time.

[0081] The example of expressing predicted quality values ​​as parametric equations defining fully differentiable surfaces or curves can offer unique technical benefits. For instance, the H.265—or High Efficiency Video Coding (HEVC)—video compression standard uses algorithms that optimize bitrate, video quality, and coding complexity. These algorithms can be sensitive to small changes in the target bitrate (where even small changes in bitrate can lead to unpredictable fluctuations in the encoded video quality). However, because predicted quality values ​​(such as bitrate) can be generated as fully differentiable surfaces or curves, local minima or maxima in the predicted quality values ​​are eliminated as the bitrate increases or decreases. This reduces unstable responses caused by large changes in bitrate, thereby significantly improving the overall stability of implementations that encode target video using the H.265 codec.

[0082] The systems and methods described in this specification offer various technical benefits for a wide range of practical applications. For example, a bitrate ladder comprises a set of encoded video streams with different bitrates and resolutions encoded from the same source video data object. Bitrate ladders are used in adaptive streaming techniques to provide optimal video quality for each viewer. Bitrate ladders enable streaming video data to each device and provide optimal encoding settings for that device. The systems and methods described in this specification allow for the simultaneous computation of encoding parameters for the entire bitrate ladder. For example, a machine learning model can generate predicted quality values ​​that define two surfaces. One surface can be quality as a function of QP and resolution, and the second surface can be bitrate as a function of QP and resolution. To determine which set of encoding parameters to use for a given ladder sequence, an isobitrate curve (which can be all encoding parameters that achieve the target bitrate) can be obtained from the second surface. The isobitrate curve can be superimposed on the predicted quality surface, and the point with the highest quality can be selected. Alternatively, the encoding parameters can be found by solving for the intersection of the isobitrate curve and the predicted quality surface. The encoding parameters corresponding to that point can be selected as the target encoding parameters. Because multiple encoding parameters can be found on the surface for multiple target bitrates, the process can be completed simultaneously for multiple target bitrates in the ladder (e.g., 10 Mbps, 1 Mbps, and / or 100 Kbps) to obtain the encoding parameters for the entire bitrate ladder without repeating the procedure for each target bitrate.

[0083] In various embodiments, the video processing platform 104 can be implemented as part of an active coding platform. The active coding platform can include a system that processes and encodes digital video packets into various formats and bitrates for streaming, broadcasting, or storage. The active coding platform can continuously encode and optimize content in real-time or near real-time to meet the specific requirements of various playback devices and / or network conditions. Therefore, the video processing platform 104 can continuously generate training data based on the video packets being continuously encoded and analyzed, and the machine learning platform 102 can continuously adjust and / or retrain the machine learning model based on the continuously updated training data. In some embodiments, some or all of the reference video packets used (e.g., at 202) to generate the training data can be pulled from the active coding platform (instead of from a reference video database). In this example, the source video data, the encoded video data, and / or the quality values ​​comparing the source video data with the encoded video data are preprocessed and do not need to be generated during process 200. By eliminating the need to encode video data and generate quality values ​​during the execution of process 200, this implementation further reduces computational requirements and increases computational throughput.

[0084] In the example embodiment, for NTraining data is generated from a reference video data packet (e.g., through machine learning training module 120). Randomly generated first and second encoding parameters (e.g., quantization parameters and resolution) are used. M Each combination encodes a video packet. Encoding parameters can be specified by the user (e.g., via a graphical user interface). M For each unique combination of a randomly generated first and second encoding parameters, (i) the bitrate of the encoded video data packet. R Recorded as (in, Indicates a specific first encoded parameter and (i) indicating specific second encoding parameters), and (ii) the encoded video data packets are decoded and analyzed to generate quality values. D This quality value was recorded as In the first encoding parameter and the second encoding parameter M Each of the combinations generates a code rate R and quality value D Next, select the optimal combination of encoding parameters. To achieve the target bit rate r or target quality value d In choosing To achieve the target bit rate r In the example, then select As a result, the bitrate is less than or equal to the target bitrate. r At the same time, it has the highest quality value d Encoding parameter set m In choosing To achieve the target quality value d In the example, then select As a result of achieving a quality value that meets or exceeds the target quality value d At the same time, it has the lowest bit rate r Encoding parameter set m .

[0085] After generating training data (e.g., by machine learning training module 120), backpropagation is used to train a machine learning model (such as a neural network) to predict the optimal encoding parameters for each reference video packet in the training data. For example, during training, the video tensor corresponding to each reference video data packet is fed into the machine learning model to generate the optimal encoding parameters for prediction. The generated optimal coding parameters will be used for prediction. With the optimal encoding parameters (from the training dataset of this reference video data packet) Comparisons are made to generate errors (e.g., and The difference between the target video and the target video is calculated, and the parameters of the machine learning model are adjusted until the error is minimized. During inference (after training), the video tensor corresponding to the target video is input into the machine learning model (e.g., at machine learning module 122), and the machine learning model generates the optimal encoding parameters for predicting the target video. Then, the optimal encoding parameters are predicted. The target video data packet is encoded (e.g., at video encoding module 124).

[0086] The following paragraphs provide examples of systems, methods, and apparatus implemented in accordance with this specification.

[0087] Example 1 includes a computer-implemented method for encoding video data based on predicted quality values ​​generated through machine learning. The method includes: feeding a target video data packet to a neural network to generate a plurality of predicted quality values ​​for the target video data packet, each of the plurality of predicted quality values ​​being associated with a different set of target coding parameters from a range of coding parameters, the neural network being trained using training data including a plurality of reference video data packets and a reference quality value computed for each reference video data packet, wherein each reference video data packet is encoded based on a different set of reference coding parameters from the range of coding parameters; setting target coding parameters for the target video data packet based on the plurality of predicted quality values; and sending a control signal to an encoder module to encode the target video data packet using the target coding parameters.

[0088] Example 2 includes the subject matter as described in Example 1, and further specifies that the plurality of predicted quality values ​​include: a first surface that defines a first predicted quality value as a function of coding parameters; and a second surface that defines a second predicted quality value as a function of coding parameters, and setting target coding parameters for the target video data packet based on the plurality of predicted quality values ​​includes: selecting coding parameters that maximize the first predicted quality value while satisfying a quality threshold for the second predicted quality value.

[0089] Example 3 includes a subject as described in any one of Examples 1 to 2, and further specifies that the predicted quality value includes at least one of Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Multi-Scale Structural Similarity Index (MS-SSIM), Video Quality Metric (VQM), Video Multi-Method Evaluation Fusion (VMAF) score, and bitrate.

[0090] Example 4 includes the subject matter as described in any one of Examples 1 to 3, and further specifies that the target encoding parameters include at least one of the following: encoding codec, quantization parameter (QP), output bitrate, and output resolution.

[0091] Example 5 includes the subject matter as described in any one of Examples 1, 3 and 4, and further specifies that the plurality of predicted quality values ​​are represented as coefficients of a predefined parametric equation.

[0092] Example 6 includes the subject matter as described in Example 5, and further specifies that the plurality of predicted quality values ​​include independent variables of the parametric equation, wherein the independent variables represent encoding parameters.

[0093] Example 7 includes the subject matter as described in Example 6, and further specifies that: the plurality of predicted quality values ​​include the dependent variable of the parametric equation, and wherein the dependent variable represents the predicted quality value.

[0094] Example 8 includes the subject matter as described in Example 7, and further specifies that the parametric equations define differentiable curves.

[0095] Example 9 includes the subject matter as described in Example 7, and further specifies that the parametric equations define differentiable surfaces.

[0096] Example 10 includes the subject matter as described in any one of Examples 1 to 9, and further specifies that the different reference coding parameter sets include randomized coding parameter sets.

[0097] Example 11 includes the subject matter as described in any one of Examples 1 to 10, and further specifies that the training data is generated by encoding each of the plurality of reference video data packets using each different set of encoding parameters.

[0098] Example 12 includes the subject as described in any one of Examples 1 to 11, and further specifies: training the machine learning model.

[0099] Example 13 includes the subject matter as described in any one of Examples 1 to 12, and further specifies that: metadata of the target video data packet is provided to the machine learning model, the metadata including at least one of file type, selected target encoding parameters, expected playback software, and expected playback device type.

[0100] Example 14 includes the subject matter as described in any one of Examples 1 to 13, and further specifies that the plurality of reference video data packets are generated by an active coding platform.

[0101] Example 15 includes the subject matter as described in any one of Examples 1 to 14, and further specifies that: an actual quality value is calculated by comparing the encoded target video data packet with the target video data packet, and the machine learning model is retrained using the actual quality value.

[0102] Example 16 includes the subject matter as described in any one of Examples 1 to 15, and further specifies that the machine learning model includes a first machine learning model configured to generate a plurality of predicted first quality values ​​and a second machine learning model configured to generate a plurality of predicted second quality values.

[0103] Example 17 includes a subject as described in any one of Examples 1 to 16, and further specifies that the neural network includes an input layer and a plurality of subsequent layers, each of which has a width and height smaller than the preceding layer, and each of which has a depth greater than the preceding layer.

[0104] Example 18 includes the subject matter as described in any one of Examples 1 to 18, and further specifies that training data is generated repeatedly, and the machine learning model is repeatedly retrained using the training data.

[0105] Example 19 includes a non-transitory computer-readable medium comprising executable instructions that, when executed by an electronic processor of a machine, cause the machine to perform the method as described in any one of Examples 1 to 18.

[0106] Example 20 includes a system comprising: memory hardware configured to store instructions; and processor hardware configured to execute the instructions. The instructions include: providing a target video data packet to a neural network to generate a plurality of predicted quality values ​​for the target video data packet, each of the plurality of predicted quality values ​​being associated with a different set of target coding parameters from a range of coding parameters, the neural network being trained using training data including a plurality of reference video data packets and a reference quality value computed for each reference video data packet, wherein each reference video data packet is encoded according to a different set of reference coding parameters from the range of coding parameters; setting target coding parameters for the target video data packet based on the plurality of predicted quality values; and sending a control signal to an encoder module to encode the target video data packet using the target coding parameters.

[0107] The foregoing description is illustrative in nature only and does not limit the scope of this disclosure or its application. The broad teachings of this disclosure can be implemented in many different ways. While this disclosure includes specific examples, other modifications will become apparent upon examination of the accompanying drawings, the text of this specification, and the following claims. In the written description and claims, one or more steps within any given method may be performed in a different order, or steps may be performed simultaneously or omitted, without altering the principles of this disclosure. Similarly, instructions stored in a non-transitory computer-readable medium may be performed in a different order, or simultaneously or omitted, without altering the principles of this disclosure. Unless otherwise stated, the numbering or other designations of instructions or method steps are for convenience of reference and do not necessarily indicate a fixed order or sequence.

[0108] Unless explicitly stated otherwise, the context of the articles “a,” “an,” and “the” should not be interpreted as meaning “only one.” Rather, these articles should be interpreted as meaning “at least one” or “one or more.” Similarly, when the terms “the” or “said” are used to refer to a noun previously introduced by the indefinite articles “a” or “an,” unless explicitly stated otherwise, the context of the terms “the” or “said” should be similarly interpreted as meaning “at least one” or “one or more.”

[0109] Spatial and functional relationships between components—such as modules—are described using terms such as (but not limited to) “connection,” “joint,” “interface connection,” and / or “coupling.” Relationships between components can be direct or include intermediate components unless explicitly stated as “direct.” The phrase “at least one of A, B, and C” should be interpreted as indicating a logical relationship (A or B or C), where either is a non-exclusive logical OR, and should not be interpreted as meaning “at least one of A, at least one of B, and at least one of C.” The term “set” does not necessarily exclude an empty set. For example, the term “set” can have zero elements. The term “subset” does not necessarily require an appropriate subset. For example, a “subset” of set A can be co-extended with set A or include elements of set A. Furthermore, the term “subset” does not necessarily exclude an empty set.

[0110] In the accompanying drawings, the direction of the arrows typically indicates the flow of information—such as a data flow or a command flow. However, the direction of the arrow does not mean that information cannot be transmitted in the reverse direction. For example, when information is sent from the first element to the second element, the arrow may point from the first element to the second element. However, the second element may send a request for data to the first element and / or send an acknowledgment of information receipt to the first element.

[0111] Throughout this application, the term "module" or "controller" may be replaced by the term "circuit." "Module" may refer to processor hardware that executes code and memory hardware that stores the code executed by the processor hardware, as part of or including the processor hardware and memory hardware. The term "module" may include one or more jamming circuits. In various embodiments, the jamming circuit may implement a wired or wireless interface connected to or as part of a communication system. A module may use the jamming circuit to communicate with other modules. In various embodiments, the functionality of a module may be distributed among multiple modules connected via a communication system. For example, functionality may be distributed across multiple modules by a load balancing system. In various embodiments, the functionality of a module may be split among multiple computing platforms connected by a communication system.

[0112] The term "code" can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, data structures, and / or data objects. The term "memory hardware" can be a subset of the term "computer-readable medium." The term "computer-readable medium" does not cover transient electrical or electromagnetic signals or electromagnetic signals propagating through a medium—such as on an electromagnetic carrier wave. The term "computer-readable medium" is considered tangible and non-transient. The modules, methods, and apparatus described in this application can be implemented, partially or entirely, by a special-purpose computer created by configuring a general-purpose computer to perform one or more specific functions described in the computer program. The aforementioned function blocks, flowchart elements, and message sequence diagrams serve as software specifications that can be translated into computer programs through the routine work of a skilled technician or programmer.

[0113] It should also be understood that while some of the accompanying drawings illustrate hardware and software located within a particular device, these descriptions are for illustrative purposes only. In some embodiments, the illustrated components may be combined or divided into separate software, firmware, and / or hardware. For example, logic and processing may be distributed among multiple electronic processors, rather than residing within a single electronic processor and executed by that single processor. Regardless of how they are combined or divided, hardware and software components may reside on the same computing device or may be distributed among different computing devices—such as computing devices interconnected via one or more networks or other communication systems.

[0114] In the claims, if the apparatus or system is claimed to include an electronic processor or other elements configured in a certain way, then the claim or the claimed element shall be interpreted as referring to one or more electronic processors (or other elements, as appropriate). If the electronic processor (or other element) is described as being configured to perform one or more determinations or execute one or more steps, then the claim shall be interpreted as referring to any combination of the one or more electronic processors (or any combination of the one or more other elements) that can be configured to perform the one or more determinations (or one or more steps).

Claims

1. A computer-implemented method for encoding video data based on predicted quality values ​​generated through machine learning, the method comprising: A target video data packet is provided to a neural network to generate multiple predicted quality values ​​for the target video data packet, each of which is associated with a different set of target coding parameters from a range of coding parameters. The predicted quality values ​​are expressed by parametric equations that define fully differentiable surfaces or curves, each parametric equation representing a different type of predicted quality value. The neural network is trained using training data that includes multiple reference video data packets and reference quality values ​​computed for each reference video data packet, wherein each reference video data packet is encoded according to a different set of reference coding parameters from the range of coding parameters. The target encoding parameters of the target video data packet are set based on the multiple predicted quality values. as well as A control signal is sent to the encoder module to encode the target video data packet using the target encoding parameters.

2. The method of claim 1, wherein: The plurality of predicted quality values ​​include: A first surface, wherein the first predicted quality value is defined as a function of the encoding parameters, and The second surface defines the second predicted quality value as a function of the encoding parameters; and Setting the target encoding parameters of the target video data packet based on the multiple predicted quality values ​​includes: Choose encoding parameters that maximize the first predicted quality value while satisfying the quality threshold of the second predicted quality value.

3. The method according to any one of claims 1 to 2, wherein, The predicted quality values ​​include at least one of the following: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Multiscale Structural Similarity Index (MS-SSIM), Video Quality Metric (VQM), Video Multi-Method Evaluation Fusion (VMAF) score, and bitrate.

4. The method according to any one of claims 1 to 3, wherein, The target coding parameters include at least one of the following: codec, quantization parameter (QP), output bit rate, and output resolution.

5. The method according to any one of claims 1, 3, and 4, wherein, The multiple predicted quality values ​​are generated using the coefficients of a predefined parametric equation.

6. The method of claim 5, wherein, The plurality of predicted quality values ​​are generated using the independent variables of the parametric equation, wherein the independent variables represent encoding parameters.

7. The method of claim 6, wherein, The plurality of predicted quality values ​​are generated using the dependent variable of the parametric equation, wherein the dependent variable represents the coding parameters.

8. The method of claim 7, wherein, The parametric equations define differentiable curves.

9. The method of claim 7, wherein, The parametric equations define differentiable surfaces.

10. The method as claimed in any of the preceding claims, wherein, The different reference coding parameter sets include randomized coding parameter sets.

11. The method of any of the preceding claims, further comprising generating the training data by encoding each of the plurality of reference video data packets using each of a different set of encoding parameters.

12. The method of any of the preceding claims, further comprising training the machine learning model.

13. The method of any preceding claim, further comprising providing the machine learning model with metadata of the target video data packet, the metadata including at least one of file type, selected target encoding parameters, expected playback software, and expected playback device type.

14. The method as claimed in any of the preceding claims, wherein, The multiple reference video data packets are generated by an active encoding platform.

15. The method of any of the preceding claims, further comprising calculating an actual quality value by comparing an encoded target video data packet with the target video data packet, and retraining the machine learning model using the actual quality value.

16. The method as claimed in any of the preceding claims, wherein, The machine learning model includes a first machine learning model configured to generate multiple predictions of a first quality value and a second machine learning model configured to generate multiple predictions of a second quality value.

17. The method as claimed in any of the preceding claims, wherein: The neural network includes an input layer and multiple subsequent layers; Each subsequent layer is smaller in width and height than the previous layer; and The depth of each subsequent layer is greater than that of the previous layer.

18. The method as claimed in any of the preceding claims, wherein: Training data is generated repeatedly, and The machine learning model is repeatedly retrained using the training data.

19. A non-transitory computer-readable medium comprising executable instructions that, when executed by an electronic processor of a machine, cause the machine to perform the method as described in any of the preceding claims.

20. A system comprising: Memory hardware configured to store instructions; as well as Processor hardware configured to execute the instructions, wherein the instructions include: A target video data packet is provided to a neural network to generate multiple predicted quality values ​​for the target video data packet. Each of the multiple predicted quality values ​​is associated with a different set of target coding parameters from a range of coding parameters. The predicted quality values ​​are expressed by parametric equations defining fully differentiable surfaces or curves, each parametric equation representing a different type of predicted quality value. The neural network is trained using training data including multiple reference video data packets and reference quality values ​​computed for each reference video data packet, wherein each reference video data packet is encoded according to a different set of reference coding parameters from the range of coding parameters. The target encoding parameters of the target video data packet are set based on the multiple predicted quality values, and A control signal is sent to the encoder module to encode the target video data packet using the target encoding parameters.

Images