Machine learning techniques for video downsampling

Abstract

In various embodiments, a training application trains a convolutional neural network to downsample an image in a video encoding pipeline. The convolutional neural network comprises at least two residual blocks and is associated with a downsampling scale factor. The training application runs the convolutional neural network on an original image to generate a downsampled image. The training application then runs an upsampling algorithm on the downsampled image to generate a reconstructed image having an equal resolution to the original image. The training application calculates a reconstruction error based on the reconstructed image and the original image. The training application updates at least one parameter of the convolutional neural network based on the reconstruction error to generate a trained convolutional neural network. Advantageously, implementing the trained convolutional neural network in a video encoding pipeline can mitigate image quality degradation typically experienced in conventional video encoding pipelines implementing conventional downsampling techniques.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Patent Application No. 17 / 133,206, filed December 23, 2020, the contents of which are incorporated herein by reference. [Technical Field]

[0002] Various embodiments of the present disclosure relate generally to computer science and video coding techniques, and more particularly to machine learning techniques for downsampling video. [Background technology]

[0003] A typical video streaming service provides users with access to a library of media titles that can be viewed on a variety of client devices. During operation, a given client device connects to the video streaming service under varying connection conditions and is therefore susceptible to changes in available network bandwidth. Therefore, various codestreams of a single media title are provided to the client device so that the media title can be streamed to the client device without interruption, regardless of the available network bandwidth. When available network bandwidth is relatively low, a "lower-quality" codestream is typically streamed to the client device, and when available network bandwidth is relatively high, a "higher-quality" codestream is typically streamed to the client device.

[0004] To generate different codestreams for a given media title, video streaming services typically encode the media title multiple times using a video encoding pipeline. The video encoding pipeline removes different amounts of information from the original video associated with a given media title to generate multiple encoded videos, each associated with a different bitrate. In practice, a typical encoding pipeline downsamples the original video to multiple lower resolutions and then compresses the original video and each downsampled lower resolution video with different compression levels to generate the different encoded videos. If the bitrate associated with a particular encoded video exceeds or equals the available network bandwidth, the encoded video can be streamed to a client device without interruption.

[0005] To play a given media title on the client device, an endpoint application can be implemented on the client device. The endpoint application is configured to select from various different encoded videos generated for the media title based on available network bandwidth. Once the endpoint application selects a given encoded video, one or more discontinuous pieces (i.e., "chunks") of the selected encoded video are streamed to the client device for playback. Upon receiving the selected encoded video chunk, the endpoint application decodes the chunk and then, optionally, upsamples the decoded chunk to generate a corresponding reconstructed video chunk. The reconstructed video has a resolution equal to the client device's display. The endpoint application then plays the various reconstructed video chunks to simulate playback of the media title on the client device.

[0006] Many video encoding pipelines implement "lossy" encoding, which involves removing information from the original video of a media title to generate various encoded videos of the media title. Thus, the image quality of a reconstructed video chunk is typically lower than the image quality of its corresponding original video chunk, and the reconstructed video chunk typically contains more image quality loss and artifacts. Furthermore, a reduction in the bitrate associated with an encoded video chunk typically results in a corresponding reduction in image quality of the reconstructed video chunk.

[0007] One drawback of conventional video coding pipelines is that the sampling algorithms typically implemented in the downsamplers within the pipeline are designed to be robust across a wide variety of digital signal types, rather than being specifically designed for a specific type of digital signal. For example, downsamplers are rarely designed to minimize the degradation of the reconstructed video due to a reduction in the resolution of the original video. This can result in a lower overall quality of the reconstructed video associated with a given encoding bitrate, potentially resulting in a suboptimal image quality. Conversely, achieving a desired target quality level for the reconstructed video can unnecessarily increase the total number of bits used to encode the original video. Summary of the Invention [Problem to be solved by the invention]

[0008] As can be seen from the above, there is a need in the art for more effective techniques for downsampling source video in video coding pipelines. [Means for solving the problem]

[0009] In one embodiment, a method for training a neural network to downsample an image in a video encoding pipeline is provided, the method including: executing a first convolutional neural network, the first convolutional neural network having at least two residual blocks and associated with a first downsampling scale factor, on a first original image having a first resolution to generate a first downsampled image; executing an upsampling algorithm on the first downsampled image to generate a first reconstructed image having the first resolution; calculating a first reconstruction error based on the first reconstructed image and the first original image; and updating at least one parameter of the first convolutional neural network based on the first reconstruction error to generate a trained convolutional neural network.

[0010] At least one of the technical advantages of the disclosed technology over the prior art is that, when implemented in a video coding pipeline, the technology can mitigate the degradation in image quality typically experienced in conventional video coding pipelines. To achieve this, a downsampling convolutional neural network implemented in the video coding pipeline is trained to reduce reconstruction error when generating downsampled image representations of original images during the video coding process. The reconstruction error approximates the degradation in image quality that occurs in reconstructed image frames due to a reduction in the resolution of the corresponding original image frames. Therefore, the disclosed technology can improve the image quality of reconstructed images for a given bitrate compared to prior art. Conversely, the disclosed technology can reduce the number of bits used to encode the original image to achieve a given target image quality compared to prior art. These technical advantages result in one or more technical improvements over prior art approaches.

[0011] Having briefly summarized the concept of the present invention above, the concept of the present invention will now be more particularly described with reference to various embodiments, some of which are also illustrated in the accompanying drawings, so that the features of the various embodiments described above can be understood in detail. However, it should be noted that the accompanying drawings merely illustrate exemplary embodiments of the concept of the present invention and therefore should not be considered to limit the scope of the present disclosure in any way, as there may be other embodiments having similar effects. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a conceptual diagram of a system configured to implement one or more aspects of various embodiments. [Figure 2] FIG. 2 illustrates one of the training applications shown in FIG. 1 in more detail, according to various embodiments. [Figure 3] FIG. 2 is a more detailed diagram illustrating downsampling by a non-integer factor by one of the trained downsampling convolutional neural networks shown in FIG. 1 , according to various embodiments. [Figure 4] FIG. 2 is a more detailed diagram illustrating integer downsampling by one of the trained downsampling convolutional neural networks shown in FIG. 1 , according to various embodiments. [Figure 5] FIG. 1 is a flow diagram illustrating method steps for generating a trained down-sampling convolutional neural network, according to various embodiments. [Figure 6] 1 is a flow diagram illustrating method steps for encoding source video, according to various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0013] In the following description, numerous specific details are set forth to provide a more thorough understanding of various embodiments. However, it will be apparent to one skilled in the art that the concepts of the present invention may be practiced without one or more of these specific details.

[0014] To stream a media title to any number of client devices without interruption, typical video streaming services pre-generate multiple different encoded videos based on the original video associated with the media title. To reduce the number of bits used by the encoded videos representing the original video, video streaming services typically use video encoding pipelines that remove selected information from the original video when pre-generating the encoded videos. As the amount of information removed increases when generating the encoded videos, the quality of the reconstructed video that can be generated from the encoded videos to approximate the original video typically decreases. Therefore, the balance between bit rate and the quality of the reconstructed video corresponding to the encoded videos varies for each encoded video.

[0015] Generally, when available network bandwidth is relatively low, encoded video with a low bit rate and low quality is typically streamed to the client device, while when available network bandwidth is relatively high, encoded video with a high bit rate and high quality is typically streamed to the client device. In operation, as the encoded video is streamed to the client device, the client device generates a reconstructed video based on the encoded video and then plays the reconstructed video to simulate playback of the media title.

[0016] To generate coded videos with various bit rates, a video coding pipeline typically implements a downsampler and an encoder. The downsampler typically operates on an original image. Thus, in a video coding pipeline, the downsampler is configured to perform a downsampling process on each frame of the original image to generate, for each frame, multiple low-resolution image frames with different resolutions. The encoder then compresses the original image and each low-resolution image with different compression levels to generate various coded videos.

[0017] One drawback of conventional video coding pipelines is that the downsamplers in the pipelines are rarely designed to minimize the degradation of the quality of the reconstructed video due to a reduction in the resolution of the source video. As a result, for a given bitrate, the associated reconstructed video may have a lower overall quality and may not be of optimal quality. Conversely, achieving a target quality level for the reconstructed video may unnecessarily increase the number of bits used to encode the source video.

[0018] In contrast, in the techniques disclosed herein, a training application uses machine learning techniques to generate trained downsampling convolutional neural networks (CNNs), which reduce reconstruction errors when downsampling an original image. Each reconstruction error correlates to the end-to-end loss in image quality between the original image and its corresponding reconstructed image. In some embodiments, each trained downsampling CNN is associated with a different downsampling factor. Each trained downsampling CNN maps the original image to a downsampled image representation that has a lower resolution than the original image by the downsampling factor associated with the trained downsampling CNN.

[0019] To generate a trained downsampling CNN corresponding to a given downsampling factor, the training application first generates an untrained version of the downsampling CNN based on the downsampling factor. As those skilled in the art will recognize, CNNs are typically limited to integer-based downsampling operations. Therefore, problems can arise when attempting to downsample an original image based on a non-integer factor using a CNN. For example, with conventional CNN technology, it is difficult to downsample a 1920 x 1080 original video frame by 1 / 1.5 to generate a corresponding 1280 x 720 low-resolution video frame.

[0020] To address the above problem, the training application generates a downsampling CNN that includes, but is not limited to, two residual blocks. As used herein, a "residual block" is a portion of a CNN that generates the output of the residual block by mapping the input to the residual block to a residual and adding the resulting residual to a function of the input of the residual block. Each residual block can be trained to map the input to the residual block to a residual, thereby adjusting the output of the residual block.

[0021] If the downsampling factor is a non-integer, the training application sets the resampling factor numerator and the resampling factor denominator to be equal to the numerator and denominator, respectively, of an irreducible fraction representing the downsampling factor. The training application then generates a downsampling CNN, including (but not limited to) an upsampling residual block followed by a downsampling residual block. The upsampling block upsamples the original image by the resampling factor numerator to generate an upsampled image. The downsampling block then downsamples the upsampled image to generate a downsampled image representation with a resolution that is the downsampling factor lower than the original image. For example, a downsampling CNN associated with a downsampling factor of 1.5 upsamples a 1920x1080 original image by an integer factor of 2 to generate a 3840x2160 upsampled image. The downsampling CNN then downsamples the 3840x2160 upsampled image by an integer factor of 3 to generate a 1280x720 downsampled image representation.

[0022] On the other hand, if the downsampling factor is an integer, the training application generates a downsampling CNN that includes (but is not limited to) an identity residual block followed by a downsampling residual block. The identity residual block is a degenerate version of the upsampling residual block and does not perform resampling. The training application uses the downsampling factor as the denominator of the resampling factor when setting the downsampling residual block.

[0023] The training application then trains the downsampling CNN based on a training image database to generate a trained downsampling CNN. The training image database may include, but is not limited to, any number and / or types of training images (e.g., frames from a training video). In operation, the training application divides each training image into multiple portions, referred to herein as "image patches." The training application then performs any number and / or types of training operations on the downsampling CNN based on the image patches to generate a trained downsampling CNN.

[0024] In some embodiments, a training application iteratively trains a downsampling CNN to reduce the reconstruction error associated with an image patch. To calculate the reconstruction error for a given image patch, the training application first inputs the image patch to the downsampling CNN. In response, the downsampling CNN outputs a downsampled image representation of the selected image patch. The training application then generates a reconstructed image corresponding to the image patch by upsampling the downsampled image representation by a downsampling factor. The training application sets the reconstruction error for the image patch to be equal to the mean squared error of the reconstructed image for the image patch. The training application then updates the parameters of the downsampling CNN so that the reconstruction error can be mitigated by the residual generated by the updated downsampling CNN.

[0025] When the training application finishes training a downsampling CNN associated with a certain downsampling factor, the training application saves the downsampling CNN as a trained downsampling CNN corresponding to that downsampling factor. The training application can transmit the trained downsampling CNN to any number of other software applications. Note that in some embodiments, the training application is configured to generate multiple trained downsampling CNNs, each corresponding to a different downsampling factor, so that the trained downsampling CNN can be used in place of a conventional downsampler in a video coding pipeline.

[0026] At least one of the technical advantages of the disclosed technology over the prior art is that by implementing the disclosed technology in a video coding pipeline, the degradation in image quality typically experienced in conventional video coding pipelines can be mitigated. In particular, by implementing a trained downsampling CNN in a video coding pipeline, the degradation in image quality that occurs in the corresponding reconstructed image frames due to a reduction in the resolution of the original image frames can be reduced. As a result, the image quality of the reconstructed image for a given bitrate can be improved compared to when using a conventional video coding pipeline. Conversely, compared to a conventional coding pipeline, the video coding pipeline can use fewer bits to encode the original image to achieve a given target image quality. For example, experimental results have shown that using a trained downsampling CNN in a video coding pipeline instead of a conventional downsampler can reduce the number of bits used to encode a given original image by nearly 25% to achieve a given target image quality. Another technical advantage of the techniques of the present disclosure is that, despite constraints on the implementation of the downsampling process in the CNNs, any number of the trained downsampling CNNs can downsample the original image, and therefore the original video, by a non-integer downsampling factor. These technical advantages provide one or more technical improvements over prior art approaches.

[0027] System Overview 1 is a conceptual diagram of a system 100 configured to implement one or more aspects of various embodiments. For ease of explanation, multiple instances or versions of a similar object are indicated with a reference number that identifies the object and, where necessary, alphanumeric characters in parentheses that identify the instance or version. As shown, in some embodiments, system 100 includes, but is not limited to, compute instance 110(1), compute instance 110(2), training image database 122, bitrate ladder 162, client device 180, and content delivery network ("CDN") 170.

[0028] In some other embodiments, system 100 may omit any number of compute instances 110(2)-110(3), bitrate ladder 162, client devices 180, and CDN 170, or any combination thereof. In some embodiments, system 100 may include, without limitation, any combination of any number of compute instances 110, any number of training image database 122, any number (including zero) of bitrate ladder 162, any number (including zero) of client devices 180, and any number of CDN 170. These components of system 100 may be distributed across any number of common geographic locations and / or any number of different geographic locations and / or may be implemented in one or more cloud computing environments (i.e., encapsulated shared resources, software, data, etc.) in any combination.

[0029] As shown, compute instance 110(1) includes, but is not limited to, processor 112(1) and memory 116(1), and compute instance 110(2) includes, but is not limited to, processor 112(2) and memory 116(2). Compute instance 110(1) and compute instance 110(2) may be referred to herein individually as "compute instance 110" or collectively as "compute instances 110." Processor 112(1) and processor 112(2) may be referred to herein individually as "processor 112" or collectively as "processors 112." Additionally, memory 116(1) and memory 116(2) may be individually referred to herein as "memory 116" or collectively referred to herein as "memories 116." Each compute instance 110 may be implemented in a cloud computing environment, as part of another distributed computing environment, or standalone.

[0030] The processor 112 may be any instruction execution system, apparatus, or device capable of executing instructions. For example, the processor 112 may comprise a central processing unit, a graphics processing unit, a controller, a microcontroller, a state machine, or any combination thereof. The memory 116 of the compute instance 110 stores content such as software applications and data used by the processor 112 of the compute instance 110. The memory 116 may be one or more local or remote digital storage devices and may take the form of readily available memory (e.g., random access memory, read-only memory, floppy disk, hard disk), etc.

[0031] In some other embodiments, any number of compute instances 110 may each include any combination of any number of processors 112 and any number of memories 116. In particular, any number of compute instances 110 (including one) may provide a multiprocessing environment in any manner that is technically feasible.

[0032] In some embodiments, storage (not shown) may supplement or replace the memory 116 of the compute instance 110. The storage may comprise any number and type of external memory accessible to the processor 112 of the compute instance 110. For example, without limitation, the storage may include a secure digital card, external flash memory, portable compact disk read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0033] Generally, each compute instance 110 is configured to implement one or more software applications. For purposes of explanation, each software application is described as residing in the memory 116 of one of the compute instances 110 and executing on the processor 112 of that one compute instance 110. However, in some embodiments, the functionality of each software application may be distributed across any number of other software applications, and these other software applications may be configured to reside in the memory 116 of any number of compute instances 110 and execute on the processors 112 of any number of compute instances 110, in any combination. Additionally, the functionality of any number of software applications may be integrated into a single software application.

[0034] Specifically, in some embodiments, downsampling / encoding application 150 and ladder application 160 reside in memory 116(2) of compute instance 110(2) and execute on processor 112(2) of compute instance 110(2). Note that in some embodiments, downsampling / encoding application 150 and ladder application 160 are collectively referred to as a "video encoding pipeline." In the same or other embodiments, ladder application 160 configures downsampling / encoding application 150 to generate encoded videos 148(1)-148(V) based on source video 152, where V may be any positive integer. Note that for ease of explanation, encoded videos 148(1)-148(V) may be individually referred to as "encoded videos 148" or collectively referred to as "encoded videos 148" herein.

[0035] Original video 152 may include any amount and / or type of video content, including, but not limited to, a feature film, an episode(s) of a television program, or any portion (including an entire music video), to name just a few examples, and each of encoded videos 148 may include, but is not limited to, coded video content derived from the video content included in original video 152 based on different coding points (not shown).

[0036] In some embodiments, each coding point has, but is not limited to, a resolution (not shown) and a quantization parameter (“QP”) value (not shown). In the same or other embodiments, the resolution of a given coding point specifies a resolution that is equal to or lower than the resolution of source video 152. The QP value is a coding parameter value that allows monotonic behavior with respect to bitrate and image quality when encoding video content. A higher QP value results in a lower bitrate, but at the expense of lower image quality. In some other embodiments, each coding point may include any number of coding parameter sets (not shown), each containing any number and type of coding parameters, instead of or in addition to a QP value.

[0037] In some embodiments, to generate the encoded video 148, the downsampling / encoding application 150 downsamples the original video 152 to each of multiple low resolutions specified by multiple coding points, thereby generating multiple low-resolution videos (not shown) with different resolutions. As will be recognized by those skilled in the art, downsampling the original video 152 to a predetermined resolution includes downsampling each frame (not shown in FIG. 1 ) of the original video 152 to the predetermined resolution. The downsampling / encoding application 150 encodes the original video 152 or the low-resolution video having the resolution specified at each coding point based on a set of coding parameters (e.g., a QP value) specified at each coding point, thereby generating the encoded video 148 corresponding to each coding point. In some embodiments, the downsampling / encoding application 150 stores the encoded video 148 in a memory accessible to at least one other software application. In the same or other embodiments, the downsampling / encoding application 150 also transmits the encoded video 148 to at least one other software application (e.g., a bitrate ladder).

[0038] In some embodiments, ladder application 160 generates bitrate ladder 162 based on coded video 148. Bitrate ladder 162 is designed to achieve a target image quality during playback of a media title at any instance of client device 180 based on available network bandwidth. In some embodiments, ladder application 160 selects coded video subset 164 of coded video 148 that provides the best balance between image quality and bitrate, and generates bitrate ladder 162 based on coded video subset 164.

[0039] In the same or other embodiments, bitrate ladder 162 may include, but is not limited to, any number of different bitrate-resolution pairs (not shown), each corresponding to a different coded image 148 among multiple coded images 148 included in coded image subset 164. The bitrate-resolution pair corresponding to a coded image 148 specifies the bitrate of that coded image 148 and the resolution of the original or lower resolution image that was encoded by downsampling / encoding application 150 to generate that coded image 148.

[0040] In some embodiments, ladder application 160 provides coded video subset 164 to CDN 170 and bitrate ladder 162 to any number of instances of client device 180 (including client device 180 shown in FIG. 1). In some embodiments, CDN 170 stores any amount and / or type of digital content, such as any number of copies of coded video subset 164, on any number of servers in any number of different geographic regions. In the same or other embodiments, CDN 170 transmits the digital content to any number of instances of client device 180 in response to client requests (not shown).

[0041] Client device 180 may be any type of device capable of executing endpoint application 190 and displaying any type of visual content, such as images, on any type of display device 182. For example, client device 180 may be, but is not limited to, a smart television, a game console, a desktop computer, a laptop, a smartphone, a tablet, etc. Display device 182 may be, but is not limited to, a liquid crystal display, a light emitting diode display, a projection display, a plasma display panel, etc.

[0042] In some embodiments, when a media title associated with original video 152 is played on client device 180, endpoint application 190 running on client device 180 first selects one of the bitrate-resolution pairs included in bitrate ladder 162 based on the available network bandwidth of the connection between client device 180 and CDN 170. Endpoint application 190 then sends a client request to CDN 170, requesting that CDN 170 stream at least a portion of encoded video 148 corresponding to that bitrate-resolution pair to client device 180. In response, CDN 170 streams non-contiguous pieces of encoded video 148 to client device 180 for playback. These non-contiguous pieces are referred to herein as "chunks."

[0043] For ease of explanation, Figure 1 illustrates an encoded video chunk 172 as a chunk of one encoded video 148 that is selected and streamed to a client device 180 at a particular point in time. Typically, client device 180 receives a chunk sequence of chunks of encoded video 148 over time, and therefore the specific chunk of encoded video 148 that an encoded video chunk 172 represents will change over time. For ease of explanation, different chunks of encoded video 148 will be referred to as different instances of encoded video chunk 172. Different instances of encoded video chunk 172 will be individually referred to as "encoded chunk 172" or collectively referred to as "encoded video chunks 172" herein.

[0044] In some embodiments, upon receiving encoded video chunks 172, endpoint application 190 decodes encoded video chunks 172 to generate decoded video chunks (not shown). Endpoint application 190 then performs any number and / or type of sampling (e.g., upsampling or downsampling) on ​​the decoded video chunks (including zero) to generate reconstructed video chunks 192 having the same resolution as display device 182. To simulate playback of a media title on client device 180, endpoint application 190 then plays a sequence of different instances of reconstructed video chunks 192 corresponding to different chunks of original video 152. For purposes of illustration, different chunks of the reconstructed video (not shown) are referred to as different instances of reconstructed video chunks 192. In this specification, different instances of reconstructed video chunk 192 may be referred to individually as "reconstructed video chunk 192" or collectively as "reconstructed video chunks 192."

[0045] As described above, conventional video coding pipelines typically include conventional downsamplers that perform conventional sampling algorithms to downsample source video to lower-resolution video. However, conventional sampling algorithms are rarely designed to minimize the degradation in quality of the reconstructed video resulting from a reduction in the resolution of the source video. As a result, when generating coded video using conventional video coding pipelines, for a given coding bitrate, the overall quality of the reconstructed video associated with that bitrate obtained from the coded video may be reduced, resulting in suboptimal quality. Furthermore, when encoding source video using conventional video coding pipelines, achieving a predetermined target quality level for the reconstructed video may unnecessarily increase the total number of bits used for coding.

[0046] Downsampling of original video using machine learning technology To address the above problems, the system 100 includes, but is not limited to, training applications 130(1)-130(N), where N can be any positive integer. The training applications 130(1)-130(N) use machine learning techniques to generate trained downsampling convolutional neural networks (“CNNs”) 140(1)-140(N), respectively. The trained downsampling CNNs 140(1)-140(N) reduce reconstruction error (not shown in FIG. 1 ) when downsampling source images (not shown in FIG. 1 ). Each source image can be any portion (including the entire image) of any type of image (e.g., a frame of source video 152).

[0047] In some embodiments, the reconstruction error is any type of metric that correlates to the end-to-end loss of image quality between an original image and an image reconstructed from the original image. For example, in some embodiments, the training application 130(1)-130(N) optimizes the trained downsampling CNNs 140(1)-140(N) to reduce the mean squared error of the reconstructed image relative to the original image when upsampling the original image.

[0048] Training applications 130(1)-130(N) are different instances of training application 130 (not explicitly shown). For ease of explanation, training applications 130(1)-130(N) may be individually referred to as “training application 130” or collectively referred to as “training applications 130.” As shown, in some embodiments, training application 130 resides in memory 116(1) of compute instance 110(1) and executes on processor 112(1) of compute instance 110(1). In other embodiments, each training application 130 may reside in any of memory 116 of any number of compute instances 110 and execute on any of processors 112 of any number of compute instances 110, in any combination.

[0049] In some embodiments, the training applications 130(1) through 130(N) generate trained downsampling CNNs 140(1) through 140(N), respectively, based on the corresponding downsampling factors 124(1) through 124(N) and the training image database 122. For ease of explanation, the trained downsampling CNNs 140(1) through 140(N) may be individually referred to as "trained downsampling CNNs 140" or collectively referred to as "trained downsampling CNNs 140." The downsampling factors 124(1) through 124(N) may be individually referred to as "downsampling factors 124" or collectively referred to as "downsampling factors 124."

[0050] Each downsampling factor 124 can be any integer or any non-integer. The downsampling factor 124(x), where x is an integer between 1 and N, configures the training application 130(x) to generate a trained downsampling CNN 140(x) that downsamples a given original image by the downsampling factor 124(x) to generate a downsampled image representation of the original image (not shown in FIG. 1 ). Note that the trained downsampling CNN 140(x) may also be referred to herein as the “trained downsampling CNN 140 corresponding to the downsampling factor 124(x).” The downsampled image representation is a downsampled image that represents the original image.

[0051] As described in more detail below in the description of FIG. 2 , in some embodiments, the training application 130 first generates an untrained downsampling CNN (not shown in FIG. 1 ) based on a given one of the downsampling factors 124 to generate a trained downsampling CNN 140 corresponding to the given one of the downsampling factors 124. The training application 130 then trains this downsampling CNN based on the training image database 122 to generate the trained downsampling CNN 140 corresponding to the given downsampling factor 124.

[0052] In some embodiments, when the downsampling factor 124 is a non-integer, the training application 130 generates a downsampling CNN that upsamples the original image to generate an upsampled image (not shown in FIG. 1 ) and then downsamples the upsampled image to generate a downsampled image representation. More precisely, the downsampling CNN upsamples the original image by an upsampling factor equal to the resampling factor numerator (not shown in FIG. 1 ) and downsamples the upsampled image by a factor equal to the resampling factor denominator (not shown in FIG. 1 ). The resampling factor numerator and the resampling factor denominator are equal to the numerator and denominator, respectively, of an irreducible fraction representing the downsampling factor 124. Thus, the downsampling CNN maps from the original image to a downsampled image representation having a downsampled image resolution that is lower than the resolution of the original image by the downsampling factor 124.

[0053] On the other hand, if downsampling factor 124 is an integer, training application 130 generates a downsampling CNN that maps from the original image to a relaxed original image (not shown in FIG. 1) and then maps the relaxed original image to a downsampled image representation that has the same resolution as the original image and a resolution that is reduced by downsampling factor 124.

[0054] In some embodiments, training application 130 trains the downsampling CNN based on training image database 122 to generate trained downsampling CNN 140. As shown, in some embodiments, training image database 122 includes, but is not limited to, training images 102(1) through 102(T), where T may be any positive integer. Training images 102(1) through 102(T) may be individually referred to herein as "training images 102" or collectively referred to as "training images 102."

[0055] Each training image 102 can be any type of image and can be associated with a different resolution. In some embodiments, the number of training images 102 are frames of a video (e.g., source video 152). In the same or other embodiments, the number of training images 102 are selected to be representative of one or more image and / or video characteristics (e.g., resolution, genre, etc.) that the trained downsampling CNN 140 is to match when tuning.

[0056] As described in more detail below with respect to FIG. 2 , in some embodiments, the training application 130 iteratively trains a downsampling CNN to reduce the reconstruction error when downsampling discontinuous portions of the training image 102 (so-called “image patches”). To calculate the reconstruction error for a given image patch, the training application 130 first inputs the image patch into the downsampling CNN. In response, the downsampling CNN generates a downsampled image representation of the image patch. The training application 130 configures a differentiable upsampling algorithm (not shown) to generate a reconstructed image by upsampling the downsampled image representation by the downsampling factor 124. The training application 130 then calculates the reconstruction error based on the image patch and the reconstructed image in any technically feasible manner. For example, in some embodiments, the training application 130 configures the reconstruction error to be equal to the mean squared error of the reconstructed image for the image patch.

[0057] Once training application 130 finishes training the downsampling CNN, training application 130 saves the downsampling CNN as trained downsampling CNN 140, which corresponds to downsampling factor 124. For ease of explanation, FIG. 1 italics some example values ​​associated with trained downsampling CNN 140(1) and trained downsampling CNN 140(2). As shown, in some embodiments, downsampling factor 124(1) is 1.5. Thus, training application 130(1) upsamples the original image by a factor of 2 (denoted by "↑2" in the figure) to generate an upsampled image, and then downsamples the upsampled image by a factor of 3 (denoted by "↓3" in the figure) to generate a downsampled image representation, creating trained downsampling CNN 140(1). In the same or other embodiments, downsampling factor 124(2) is 2. Thus, the training application 130(2) generates a trained downsampling CNN 140(2) that maps the original image to a relaxed original image (indicated by "↑1" in the figure) and then downsamples the relaxed original image by a factor of 2 (indicated by "↓2" in the figure) to generate a downsampled image representation.

[0058] In some embodiments, the training application 130 stores the trained downsampling CNN 140 in any memory accessible to at least one other software application. Also, in the same or other embodiments, the training application 130 transmits the trained downsampling CNN 140 to any number and / or type of other software applications. For example, in some embodiments, the training applications 130(1)-130(N) each transmit the trained downsampling CNN 140(1)-140(N) to the downsampling / encoding application 150.

[0059] In some embodiments, to generate encoded video 148 using the trained downsampling CNN 140, a downsampling / encoding application 150 determines a subset (not shown) of downsampling factors 124 based on the resolution of the original video 152 and the multiple resolutions specified at the multiple coding points. Then, for each of the multiple downsampling factors 124 included in the subset of downsampling factors 124, the downsampling / encoding application 150 executes the trained downsampling CNN 140 corresponding to each downsampling factor 124 on the original video 152 to generate low-resolution video corresponding to each downsampling factor 124. Then, the downsampling / encoding application 150 encodes the original video 152 or encodes the low-resolution video having the resolution specified at each coding point based on a set of encoding parameters (e.g., a QP value) specified at each coding point to generate encoded video 148 corresponding to each coding point. Then, in some embodiments, a ladder application 160 generates a bitrate ladder 162 based on the encoded video 148.

[0060] In the same or other embodiments, the endpoint application executes on the client device 180, as described above. To play at least a portion of the media title associated with the original video 152 on the client device 180, the endpoint application 190 selects one of the encoded videos 148 derived from the original video 152. The endpoint application 190 may select the encoded video 148 in any technically feasible manner. For example, in some embodiments, the endpoint application 190 makes the selection of the encoded video 148 based on the bitrate-resolution pairs included in the bitrate ladder 162 and the available network bandwidth associated with the client device 180. Once one of the encoded videos 148 is selected by the endpoint application 190, any number of the encoded video chunks 172 of the encoded video 148 are streamed to the client device 180 for playback.

[0061] As described above, upon receiving each encoded video chunk 172, endpoint application 190 decodes the encoded video chunk 172 to generate a decoded video chunk (not shown). Endpoint application 190 then performs any number and / or type of sampling (e.g., upsampling or downsampling) on ​​the decoded video chunk to generate a reconstructed video chunk 192 having the same resolution as display device 182. For example, in some embodiments, endpoint application 190 upsamples the decoded video chunk. In some other embodiments, endpoint application 190 downsamples the decoded video chunk using one of trained downsampling CNNs 140(1)-140(N). Endpoint application 190 then plays the reconstructed video chunks 192 in order to simulate playback of the media title on client device 180.

[0062] It should be noted that the reconstruction error used to optimize the trained downsampling CNN 140 approximates the loss in image quality that occurs in frames of the reconstructed video chunks 192 due to a loss in resolution of the corresponding frames of the source video 152. Therefore, using the trained downsampling CNN 140 in place of a conventional downsampler in a video coding pipeline to generate coded video 148 can mitigate the loss in image quality typically experienced in conventional video coding pipelines.

[0063] In some embodiments, the downsampling / encoding application 150 can improve the quality of the reconstructed video chunks 192 for a given bitrate by using the trained downsampling CNN 140 instead of a conventional downsampler to generate the lower resolution video, which in turn can reduce the number of bits used by the downsampling / encoding application 150 to encode the original video 152 to achieve a target quality.

[0064] It should be noted that the techniques described herein are illustrative and not limiting, and may be modified without departing from the spirit and scope of the invention broadly construed. Many modifications and variations of the functionality provided by the training application 130 and trained downsampling CNN 140 will be apparent to those skilled in the art without departing from the scope and spirit of the embodiments described herein.

[0065] It will be understood that the system 100 illustrated herein is illustrative and that variations and modifications are possible. For example, the connection topology between the various components illustrated in FIG. 1 can be modified as desired. For example, in some embodiments, the training application 130 is integrated into the video encoding pipeline. In the same or other embodiments, the ladder application 160 is omitted from the video encoding pipeline.

[0066] Generating a pre-trained downsampling CNN FIG. 2 is a more detailed diagram of one of the training applications 130 shown in FIG. 1 , according to various embodiments. Specifically, FIG. 2 illustrates training application 130(1) generating a trained downsampling CNN 140(1) based on downsampling factor 124(1) and training image database 122. Downsampling factor 124(1) can be any integer or any non-integer. As noted above in the description of FIG. 1 , training image database 122 includes, but is not limited to, training images 102(1) through 102(T), where T is any positive integer. In some embodiments, the number of training images 102 are frames from any number and / or type of training video (not shown).

[0067] As shown, in some embodiments, training application 130(1) includes, but is not limited to, configuration engine 210, reconstruction error engines 232(1) through 232(P), and iteration engine 220, where P can be any positive integer. Reconstruction error engines 232(1) through 232(P) are different instances of the same version of reconstruction error engine 232 (not explicitly shown). For ease of explanation, instances of reconstruction error engine 232, regardless of version, may be individually referred to herein as "reconstruction error engine 232" or collectively referred to as "reconstruction error engines 232."

[0068] In some embodiments, the configuration engine 210 generates a downsampling CNN 240(1) based on the downsampling factor 124(1). The downsampling CNN 240(1) is a first version of the downsampling CNN 240 (not explicitly shown). The downsampling CNN 240(1) maps from an original image (not shown in FIG. 2 ) to a downsampled image representation 248 of the original image that has a resolution that is lower than the original image by the downsampling factor 124(1). The original image can be any portion of any type of image, including the entire image. The configuration engine 210 can generate the downsampling CNN 240(1) in any technically feasible manner.

[0069] As shown, in some embodiments, configuration engine 210 includes, but is not limited to, configurable non-integer scale CNN 202, configurable integer scale CNN 204, resampling scale numerator 212(1), and resampling scale denominator 214(1). As those skilled in the art will recognize, using a CNN to resize a source image based on a non-integer scale (e.g., downsampling, upsampling, etc.) can be problematic because CNNs typically only allow pooling or striding based on integer scales. However, configurable non-integer scale CNN 202 can be advantageously configured to resize a source image based on any non-integer scale, with two configurable scales that can be set to equal any integer.

[0070] Specifically, configurable non-integer scale factor CNN 202 maps a given source image to an upsampled image (not shown in FIG. 2), and then maps the upsampled image to a downsampled image representation 248. The upsampled image has a resolution that is higher than the source image by a first configurable scale factor, and downsampled image representation 248 has a resolution that is lower than the upsampled image by a second configurable scale factor.

[0071] The configurable integer scale CNN 204 is a degenerate version of the configurable non-integer scale CNN 202, and is a simplified pre-configured version of the configurable non-integer scale CNN 202 based on the first configurable scale value being "1". Thus, the configurable integer scale CNN 204 has one configurable scale. The configurable integer scale CNN 204 maps the original image to a relaxed original image (not shown in FIG. 2) and then maps the relaxed original image to a downsampled image representation 248. The relaxed original image has the same resolution as the original image, and the downsampled image representation 248 has a resolution that is lower than the original image by this one configurable scale factor.

[0072] In some embodiments, if downsampling scaling factor 124(1) is a non-integer, configuration engine 210 sets resampling scaling factor numerator 212(1) and resampling scaling factor denominator 214(1) to be equal to the numerator and denominator, respectively, of an irreducible fraction representing downsampling scaling factor 124(1). Next, to generate downsampling CNN 240(1), configuration engine 210 sets a first configurable scaling factor and a second configurable scaling factor of configurable non-integer scaling factor CNN 202 to be equal to resampling scaling factor numerator 212(1) and resampling scaling factor denominator 214(1), respectively. Conversely, if downsampling scaling factor 124(1) is an integer, configuration engine 210 sets the only configurable scaling factor of configurable integer scaling factor CNN 204 to be equal to downsampling scaling factor 124(1) to generate downsampling CNN 240(1).

[0073] In some embodiments, configurable non-integer scale CNN 202 includes, but is not limited to, one set of parameters (e.g., weights and biases) that can be trained to correct the mapping from an original image to an upsampled image and another set of parameters that can be trained to correct the mapping from the upsampled image to a downsampled image representation 248. Similarly, configurable integer scale CNN 204 includes, but is not limited to, one set of parameters that can be trained to correct the mapping from an original image to a relaxed original image and another set of parameters that can be trained to correct the mapping from the relaxed original image to a downsampled image representation 248. Thus, whether downsampling scale factor 124(1) is an integer or not, downsampling CNN 240(1) includes, but is not limited to, two sets of trainable parameters that both affect downsampled image representation 248.

[0074] For ease of explanation, a "downsampling CNN 240" refers to any CNN trained by the training application 130 to downsample images by one of the downsampling factors 124. The resulting trained CNN may also be referred to herein as a "trained downsampling CNN 140." The training application 130(1) may train the downsampling CNN 240 in any technically feasible manner.

[0075] In some embodiments, to enable the iteration engine 220 to efficiently train the downsampling CNN 240, the configuration engine 210 configures the initial reconstruction error engine 232 to calculate a reconstruction error 260 (not explicitly shown) based on an image patch (not explicitly shown) and the downsampling CNN 240(1). The image patch 228 can be any portion of any type of image, including the entire image. For ease of explanation, instances of image patches 228 may be individually referred to as "image patches 228" or collectively referred to as "image patches 228" herein. Each image patch 228 is a different original image.

[0076] As described in further detail below, the reconstruction error 260 correlates to the degradation in image quality of the reconstructed image 258 obtained from one of the image patches 228 due to the downsampling CNN 240. In particular, in the first version of the reconstruction error engine 232, the reconstruction error 260 correlates to the degradation in image quality of the reconstructed image 258 obtained from one of the image patches 228 due to the downsampling CNN 240.

[0077] As shown in the illustrated reconstruction error engine 232(1), the initial version of the reconstruction error engine 232 includes, but is not limited to, a downsampling CNN 240(1) and an upsampling engine 250. The upsampling engine 250 can implement any differentiable upsampling algorithm in any technically feasible manner. Some non-limiting examples of differentiable upsampling algorithms include a bicubic upsampling algorithm and a bilinear upsampling algorithm. The configuration engine 210 configures the upsampling engine 250 to perform upsampling at a downsampling factor of 124(1). The configuration engine 210 can configure the upsampling engine 250 in any technically feasible manner. For example, in some embodiments, the configuration engine 210 configures the upsampling engine 250 with configurable settings (not shown) associated with the upsampling algorithm implemented by the upsampling engine 250.

[0078] For ease of explanation, some example values ​​associated with configuration engine 210 in some embodiments are italicized. As shown in italics, in some embodiments, downsampling scale factor 124(1) is equal to 1.5. Based on the non-integer value of downsampling scale factor 124(1), configuration engine 210 sets resampling scale factor numerator 212(1) to be equal to 2 and resampling scale factor denominator 214(1) to be equal to 3. Based on the resampling scale factor numerator 212(1) being equal to 2 and the resampling scale factor denominator 214(1) being equal to 3, configuration engine 210 generates downsampling CNN 240(1) by configuring configurable non-integer scale factor CNN 202, as shown in italics. Additionally, configuration engine 210 configures the bicubic algorithm in upsampling engine 250 to perform upsampling by a factor of "1.5" (shown in italics), which is the downsampling factor 124(1).

[0079] When the reconstruction error engine 232 receives one of the multiple image patches 228, it inputs the image patch 228 to the downsampling CNN 240(1). In response, the downsampling CNN 240(1) outputs a downsampled image representation 248 of the image patch 228. The reconstruction error engine 232 then inputs the downsampled image representation 248 to the upsampling engine 250. In response, the upsampling engine 250 generates a reconstructed image 258 that approximates the image patch 228. The reconstruction error engine 232 then calculates and outputs a reconstruction error 260 of the image patch 228 based on the image patch 228 and the reconstructed image 258.

[0080] Reconstruction error engine 232 may calculate reconstruction error 260 in any technically feasible manner. In some embodiments, reconstruction error engine 232 calculates reconstruction error 260 based on any number and / or type of image quality indicators and / or any number and / or type of image quality models. For example, in some embodiments, reconstruction error engine 232 sets reconstruction error 260 to be equal to the mean squared error of reconstructed image 258 with respect to image patch 228. In some other embodiments, reconstruction error engine 232 sets reconstruction error 260 to be equal to the peak signal-to-noise ratio between reconstructed image 258 and image patch 228.

[0081] In some embodiments, iteration engine 220 uses reconstruction error engine 232 to perform any number and / or types of training operations on downsampling CNN 240(1) based on training image database 122. As used herein, "training operation" refers to any number and / or types of operations used to determine new values ​​for parameters (e.g., weights and biases) of a CNN (e.g., downsampling CNN 240(1)) and any number and / or types of operations used to update the CNN's parameters to these new values.

[0082] Specifically, in some embodiments, the iteration engine 220 iteratively corrects parameters in the downsampling CNN 240(1) using instances of the reconstruction error 260 calculated by the reconstruction error engine 232 based on any number of image patches 228 from the training images 102. For ease of explanation, the instances of the reconstruction error 260 may be referred to herein individually as the "reconstruction error 260" or collectively as the "reconstruction errors 260."

[0083] In some embodiments, the iteration engine 220 generates at least one instance of the image patch 228 for each training image 102 included in the training image database 122. The iteration engine 220 may generate the image patch 228 for each training image 102 in any technically feasible manner. In some embodiments, the image patches 228 in the image patch subset associated with each training image 102 are non-overlapping.

[0084] In the same or another embodiment, iteration engine 220 divides image patches 228 derived from training images 102 into batches 222(1) through 222(B), where B can be any positive integer. For ease of explanation, batches 222(1) through 222(B) may be individually referred to as "batches 222" or collectively referred to as "batches 222." Iteration engine 220 may determine the number of batches 222 to generate and distribute image patches 228 among those batches 222 in any technically feasible manner.

[0085] For example, in some embodiments, the iteration engine 220 generates batches 222 with a total number of batches greater than one but less than the total number of image patches 228, and divides the image patches 228 approximately evenly among the batches 222. In other embodiments, the iteration engine 220 generates one instance of the batch 222 and assigns all of the image patches 228 to the one batch 222. In still other embodiments, the iteration engine 220 generates batches 222 with a total number of batches equal to the total number of image patches 228, and assigns one image patch 228 to each of the generated batches 222.

[0086] As shown, in some embodiments, iteration engine 220 iteratively trains downsampling CNN 240(1) through batch iterations 230(1) through 230(Z), where Z is equal to the product of the total number of epochs (not shown) and the total number of batches 222. For ease of explanation, batch iterations 230(1) through 230(Z) may be individually referred to as "batch iterations 230" or collectively referred to as "batch iterations 230." Iteration engine 220 may determine the total number of epochs in any technically feasible manner. For example, in some embodiments, iteration engine 220 determines the total number of epochs based at least in part on input received via a graphical user interface (not shown). In the same or other embodiments, iteration engine 220 may determine the total number of epochs based on any number and / or type of convergence criteria.

[0087] In some embodiments, the iteration engine 220 executes the epochs sequentially. Furthermore, during each epoch, the iteration engine 220 executes a batch iteration 230 for each of the multiple batches 222, one at a time. Thus, the iteration engine 220 executes batch iterations 230(1) through 230(Z) sequentially. In the same or other embodiments, during each batch iteration 230, the iteration engine 220 inputs the image patches 228 included in the batch 222 to any number of reconstruction error engines 232. This input may be sequential, simultaneous, or any combination of sequential and simultaneous. In response, the reconstruction error engines 232 calculate and output a reconstruction error 260 for each of the multiple image patches 228 included in the batch 222. This processing may be sequential, simultaneous, or any combination of sequential and simultaneous.

[0088] In the same or other embodiments, iteration engine 220 calculates iteration error 270 for batch iteration 230 based on multiple reconstruction errors 260 calculated by reconstruction error engine 232 in batch iteration 230. Iteration error 270 can be any value of any type of overall objective function (e.g., overall loss function), and iteration engine 220 can calculate iteration error 270 in any technically feasible manner. For example, in some embodiments, iteration engine 220 sets iteration error 270 to be equal to the average of reconstruction errors 260 calculated by reconstruction error engine 232 in batch iteration 230.

[0089] Then, in some embodiments, iteration engine 220 completes this batch iteration 230 by performing a parameter update process 280 according to the goal of reducing iteration error 270. In the parameter update process 280, iteration engine 220 updates any number of the parameters of downsampling CNN 240 to generate a new version of downsampling CNN 240 that is better optimized for the goal of reducing iteration error 270. Iteration engine 220 can perform any number and / or types of machine learning processes to perform parameter update process 280. For example, in some embodiments, iteration engine 220 performs any number and / or types of backpropagation processes and any number and / or types of gradient descent processes to update the parameters of the current version of downsampling CNN 240, thereby generating a new version of downsampling CNN 240.

[0090] For ease of explanation, FIG. 2 illustrates in detail an example of batch iteration 230(1) according to some embodiments. As illustrated, in some embodiments, batch 222(1) includes, but is not limited to, image patches 228(1) through 228(P), where P can be any integer. Iteration engine 220 simultaneously inputs image patches 228(1) through 228(P) to reconstruction error engines 232(1) through 232(P). Note that each of reconstruction error engines 232(1) through 232(P) includes a downsampling CNN 240(1). In response, reconstruction error engines 232(1) through 232(P) simultaneously calculate and output reconstruction errors 260(1) through 260(P). Iteration engine 220 calculates iteration error 270 based on reconstruction errors 260(1) through 260(P). Next, iteration engine 220 performs a parameter update process 280 on downsampling CNN 240(1) based on iteration error 270 to generate a new version of downsampling CNN 240 (denoted as downsampling CNN “240(2)” in FIG. 2).

[0091] In some embodiments, after each batch iteration 230 is completed, the iteration engine 220 configures the reconstruction error engine 232 so that a newly generated version of the downsampling CNN 240 is used in the next batch iteration 230. For ease of explanation, different versions of the downsampling CNN 240 are referred to herein as downsampling CNNs 240(1) to 240(Z+1), where Z is the total number of batch iterations 230. Therefore, where x is an integer between 1 and Z, in batch iteration 230(x), the iteration engine 220 calculates the iteration error 270 based on the downsampling CNN 240(x) to generate downsampling CNN 240(x+1).

[0092] As shown, after completing batch iteration 230(Z), iteration engine 220 sets trained downsampling CNN 140(1) equal to downsampling CNN 240(Z+1). In some embodiments, iteration engine 220 stores trained downsampling CNN 140(1) in memory accessible to any number of other software applications. In the same or other embodiments, iteration engine 220 also transmits trained downsampling CNN 140(1) to any number of other software applications. For example, in some embodiments, iteration engine 220 transmits trained downsampling CNN 140(1) to downsampling / encoding application 150.

[0093] Pre-trained downsampling CNN Figure 3 is a more detailed diagram illustrating downsampling by a non-integer factor by one of the trained downsampling CNNs 140 shown in Figure 1, according to various embodiments. Specifically, Figure 3 illustrates the trained downsampling CNN 140(1) of Figure 1 associated with a downsampling factor 124(1), which may be any non-integer. As noted above in the description of Figure 2, the trained downsampling CNN 140(1) is also associated with a resampling factor numerator 212(1) and a resampling factor denominator 214(1). The resampling factor numerator 212(1) and the resampling factor denominator 214(1) are equal to the numerator and denominator, respectively, of a fraction (not shown) equal to the reciprocal of the downsampling factor 124(1).

[0094] As shown, in some embodiments, the trained downsampling CNN 140(1) maps an original image 308 to a downsampled image representation 248 that has a resolution that is a downsampling factor 124(1) lower than the original image 308. The original image 308 can be any portion of any image, including the entire image, at any resolution. In some embodiments, the original image 308 is at least a portion of a frame of the original video 152.

[0095] Although not shown, in the same or other embodiments, downsampling / encoding application 150 inputs each frame in original video 152 to any number of instances of downsampling CNN 240(1), either sequentially, simultaneously, or any combination of sequential and simultaneous inputs. In response, one or more instances of trained downsampling CNN 140(1) output each frame in a lower-resolution video having a resolution that is lower than the resolution of original video 152 by downsampling factor 124(1). This lower-resolution video is sometimes referred to herein as "downsampled video."

[0096] In some embodiments, the trained downsampling CNN 140(1) includes, but is not limited to, an upsampling residual block 310 and a downsampling residual block 360. As used herein, a "residual block" is a portion of a CNN that generates the output of the residual block by mapping the input to the residual block to a residual and adding the resulting residual to a function of the input of the residual block. For purposes of illustration, each residual block includes, but is not limited to, a "primary" path that maps the input to the residual block to a residual and a "shortcut" path that computes a function of the input of the residual block.

[0097] In some embodiments, when training application 130(1) performs a training process on downsampling CNN 240(1) to generate trained downsampling CNN 140(1), both upsampling residual block 310 and downsampling residual block 360 learn residual maps that jointly reduce reconstruction error 260. As a result, the parameters learned by upsampling residual block 310 and downsampling residual block 360 mitigate the degradation in image quality associated with the data loss inherent in downsampling function 380.

[0098] As shown, in some embodiments, the input to upsampling residual block 310 is original image 308, and the output from upsampling residual block 310 is upsampled image 318. The resolution of upsampled image 318 is equal to the product of the resolution of original image 308 and resampling scale numerator 212(1). For ease of explanation, the product of the resolution of original image 308 and resampling scale numerator 212(1) may be referred to herein as the "upsampled resolution."

[0099] In some embodiments, the upsampling residual block 310 includes, but is not limited to, an upsampling function 350 and a high-resolution layer stack 312. As shown, in some embodiments, a first drop connection (unnumbered) directs the original image 308 to an input to the upsampling function 350 and an input to the high-resolution layer stack 312. The output of the upsampling function 350 is an algorithmically generated upsampled image 358 having the upsampled resolution. The output of the high-resolution layer stack 312 is an upsampling residual 348 having the upsampled resolution. Then, as indicated by the circled "+" sign, in some embodiments, a first join connection calculates the sum of the upsampling residual 348 and the algorithmically generated upsampled image 358 on a sample-by-sample basis to generate the upsampled image 318.

[0100] In some embodiments, upsampling function 350 may implement any differentiable upsampling algorithm in any technically feasible manner to generate algorithmically generated upsampled image 358 based on original image 308. Some non-limiting examples of differentiable upsampling algorithms include bicubic upsampling and bilinear upsampling. In the same or other embodiments, upsampling function 350 and upsampling engine 250 included in training application 130(1) implement the same upsampling algorithm. In some embodiments, upsampling function 350 does not include trainable parameters. Thus, upsampling function 350 included in downsampling CNN 240(1) has the same functionality as upsampling function 350 included in trained downsampling CNN 140(1).

[0101] The high resolution layer stack 312 may include, but is not limited to, any number and / or type of layers trained to map from the source image 308 to the upsampled residual 348. As shown, in some embodiments, the high resolution layer stack 312 may include, but is not limited to, a high resolution layer set 320, a pair of convolutional rectified linear unit ("ReLU") layers 330(1) through 330(K), and a convolutional layer 340(1), where K may be any positive integer.

[0102] The high resolution layer set 320 may include, but is not limited to, any number and / or type of layers performing any number and / or type of upsampling operations in any manner that is technically feasible, and may optionally include any number and / or type of other layers. For example, in some embodiments, the high resolution layer set 320 may include, but is not limited to, an upsampling layer (not shown), optionally followed by an activation layer (not shown). The input to the upsampling layer is the original image 308, and the output of the upsampling layer has the upsampled resolution.

[0103] In some embodiments, the upsampling layer is a deconvolution layer with an output stride equal to resampling scale numerator 212(1). In operation, the deconvolution layer performs a strided deconvolution on the original image 308, upsampling the original image 308 by a factor equal to resampling scale numerator 212(1). In some other embodiments, the upsampling layer is an unpooling layer. The unpooling layer can perform any type of unpooling operation on the original image 308 (e.g., nearest neighbor unpooling, maximum unpooling, etc.). In the same or other embodiments, the output of the upsampling layer is input to an activation layer, and the output of the activation layer is output from the high-resolution layer set 320. For example, in some embodiments, the activation layer is a ReLU layer that applies a rectified linear activation function to each sample in the output of the upsampling layer to generate the output of the high-resolution layer set 320.

[0104] The convolutional ReLU layer pairs 330(1) to 330(K) are instances of the convolutional ReLU layer pair 330 (not explicitly shown). For ease of explanation, in this specification, the instances of the convolutional ReLU layer pair 330 (including the convolutional ReLU layer pairs 330(1) to 330(K)) may be individually referred to as a "convolutional ReLU layer pair 330" or collectively referred to as a "convolutional ReLU layer pair 330." Each convolutional ReLU layer pair 330 includes, but is not limited to, a convolutional layer and a subsequent ReLU layer. The output of the high-resolution layer set 320 is used as input to the convolutional ReLU layer pair 330(1). Additionally, where x is an integer between 2 and K, the output of convolutional ReLU layer pair 330(x-1) is input to convolutional ReLU layer pair 330(x). As shown, in some embodiments, the output of convolutional ReLU layer pair 330(K) is input to convolutional layer 340(1), and the output of convolutional layer 340(1) is upsampled residual 348.

[0105] As shown, in some embodiments, the input to downsampling residual block 360 is upsampled image 318, and the output of downsampling residual block 360 is downsampled image representation 248. The resolution of downsampled image representation 248 is equal to the resolution of upsampled image 318 divided by resampling scale factor denominator 214(1). Therefore, the resolution of downsampled image representation 248 is also equal to the resolution of original image 308 divided by downsampling scale factor 124(1). For ease of explanation, the resolution of downsampled image representation 248 may be referred to herein as the "downsampled resolution."

[0106] In some embodiments, downsampling residual block 360 includes, but is not limited to, a downsampling function 380 and a low-resolution layer stack 362. In the same or other embodiments, a second drop connection (not labeled) routes upsampled image 318 to an input to downsampling function 380 and to an input to low-resolution layer stack 362. The output of downsampling function 380 is an algorithmically generated downsampled image 388 having the downsampled resolution. The output of low-resolution layer stack 362 is a downsampling residual 378 having the downsampled resolution. Then, as indicated by the circled "+" sign, in some embodiments, a second join connection calculates the sum of downsampling residual 378 and algorithmically generated downsampled image 388 on a sample-by-sample basis to generate downsampled image representation 248.

[0107] In some embodiments, downsampling function 380 may implement any differentiable downsampling algorithm in any technically feasible manner to generate algorithmically generated downsampled image 388 based on upsampled image 318. Some non-limiting examples of differentiable downsampling algorithms include bicubic downsampling algorithms and bilinear downsampling algorithms. In some embodiments, downsampling function 380 does not include trainable parameters. Thus, downsampling function 380 included in downsampling CNN 240(1) has the same functionality as downsampling function 380 included in trained downsampling CNN 140(1).

[0108] Low-resolution layer stack 362 may include, but is not limited to, any number and / or type of layers trained to map from upsampled image 318 to downsampled residual 378. As shown, in some embodiments, low-resolution layer stack 362 may include, but is not limited to, low-resolution layer set 370, convolutional ReLU layer pairs 330(K+1) through 330(L), and convolutional layer 340(2), where K can be any non-zero integer and L can be any integer greater than (K+1).

[0109] The low-resolution layer set 370 downsamples the upsampled image 318 by a factor equal to the resampling factor denominator 214(1). The low-resolution layer set 370 can include any number and / or types of layers performing any number and / or types of downsampling operations in any manner that is technically feasible, and can optionally include any number and / or types of other layers. For example, in some embodiments, the low-resolution layer set 370 can include, without limitation, a downsampling layer (not shown), optionally followed by an activation layer (not shown). The input to the downsampling layer is the upsampled image 318, and the output from the downsampling layer has the downsampled resolution.

[0110] In some embodiments, the downsampling layer is a convolutional layer with an input stride equal to the resampling scale denominator 214(1). In operation, the convolutional layer performs a strided convolution on the upsampled image 318, downsampling the upsampled image 318 by a factor equal to the resampling scale denominator 214(1). In some other embodiments, the downsampling layer is a pooling layer. The pooling layer can perform any type of pooling operation (e.g., max pooling, mean pooling, etc.) on the upsampled image 318. In the same or other embodiments, the output of the downsampling layer is input to an activation layer, and the output of the activation layer is the output of the low-resolution layer set 370. For example, in some embodiments, the activation layer is a ReLU layer that applies a rectified linear activation function to each sample in the output of the downsampling layer to generate the output of the low-resolution layer set 370.

[0111] Convolutional ReLU layer pairs 330(K+1) through 330(L) are instances of the convolutional ReLU layer pair 330 described above in the description of the upsampling residual block 310. The output of the low-resolution layer set 370 is input to the convolutional ReLU layer pair 330(K+1). Furthermore, where x is an integer from (K+2) to L, the output of the convolutional ReLU layer pair 330(x-1) is input to the convolutional ReLU layer pair 330(x). As shown, in some embodiments, the output of the convolutional ReLU layer pair 330(L) is input to the convolutional layer 340(2), and the output of the convolutional layer 340(2) is the downsampling residual 378.

[0112] For ease of explanation, example values ​​associated with an example embodiment of trained downsampling CNN 140(1) are italicized in Figure 3. In the example embodiment, downsampling scale factor 124(1) is 1.5, resampling scale numerator 212(1) is 2, resampling scale denominator 214(1) is 3, and source image 308 is a frame of source video 152 having a resolution of 1920x1080.

[0113] As shown, in the exemplary embodiment, upsampling function 350 implements a bicubic upsampling algorithm associated with a scaling factor of "2." In operation, upsampling function 350 performs the bicubic upsampling algorithm on original image 308 to generate algorithmically generated upsampled image 358 having a resolution of 3840 x 2160. High resolution layer set 320 includes, but is not limited to, a deconvolution layer with an output stride of "2" followed by a ReLU layer. Thus, the output of high resolution layer stack 312 is upsampled residual 348 having a resolution of 3840 x 2160. Upsampling residual block 310 then calculates the sum of algorithmically generated upsampled image 358 and upsampled residual 348 on a sample-by-sample basis to generate upsampled image 318 having a resolution of 3840 x 2160.

[0114] In the exemplary embodiment shown in FIG. 3 , downsampling function 380 implements a bicubic downsampling algorithm associated with a scaling factor of "3." In operation, downsampling function 380 performs the bicubic downsampling algorithm on upsampled image 318 to generate algorithmically generated downsampled image 388 having a resolution of 1280 x 720. Low-resolution layer set 370 includes, but is not limited to, a convolutional layer associated with an input stride of "3" followed by a ReLU layer. Thus, the output of low-resolution layer stack 362 is downsampled residual 378 having a resolution of 1280 x 720. Downsampling residual block 360 then computes the sum of algorithmically generated downsampled image 388 and downsampled residual 378 on a sample-by-sample basis to generate downsampled image representation 248 having a resolution of 1280 x 720.

[0115] As illustrated for the exemplary embodiment, in some embodiments, upsampling residual 348 and downsampling residual 378 include, but are not limited to, sample values ​​at the edges of objects depicted in original image 308, thereby reducing blur artifacts in the reconstructed image caused by downsampling function 380. As a result, reconstructed image 258 obtained from downsampled image representation 248 may have higher image quality than a conventional reconstructed image obtained from a conventional downsampled image produced by a conventional downsampler.

[0116] In some embodiments, the upsampling residual 348 and the downsampling residual 378 can mitigate any number and / or type of image quality losses typically present in conventionally reconstructed images and / or videos. Furthermore, as we progress from the high-resolution layer stack 312 to the low-resolution layer stack 362, the parameters learned in the high-resolution layer stack 312 reduce the reconstruction error 260 based on all samples included in the original image 308. As a result, the effectiveness of the trained downsampling CNN 140(1) can be improved.

[0117] Also importantly, because the trained downsampling CNN 140(1) includes both the upsampling residual block 310 and the downsampling residual block 360, the trained downsampling CNN 140(1) is able to generate downsampled image representations 248 that have a resolution that is a non-integer multiple of the original image 308.

[0118] Figure 4 is a more detailed diagram illustrating downsampling by an integer factor by one of the trained downsampling CNNs 140 shown in Figure 1, according to various embodiments. Specifically, Figure 4 illustrates the trained downsampling CNN 140(2) of Figure 1 associated with a downsampling factor 124(2), which may be any integer. As noted above in the discussion of Figure 2, in some embodiments, the architecture of the trained downsampling CNN 140(2) is a degenerate version of the architecture of the trained downsampling CNN 140(1) described above in the discussion of Figure 3.

[0119] As shown, in some embodiments, the trained downsampling CNN 140(2) maps from an original image 308 to a downsampled image representation 248 that has a resolution that is less than the original image 308 by a downsampling factor 124(2). In the same or other embodiments, the trained downsampling CNN 140(2) includes, but is not limited to, an identity residual block 410 and a downsampled residual block 360.

[0120] Identity residual block 410 is a residual block that includes an identity function in its shortcut path. In some embodiments, when training application 130(2) generates trained downsampling CNN 140(2), both identity residual block 410 and downsampling residual block 360 learn a residual map that jointly reduces reconstruction error 260. As a result, the parameters learned by identity residual block 410 and downsampling residual block 360 mitigate the degradation in image quality associated with the data loss inherent in downsampling function 380.

[0121] As shown, in some embodiments, the input to identity residual block 410 is original image 308, and the output of identity residual block 410 is relaxed original image 418. The resolution of relaxed original image 418 is equal to the resolution of original image 308. In some embodiments, identity residual block 410 includes, but is not limited to, a spare layer stack 412. In the same or other embodiments, a first drop connection routes original image 308 to either the input to spare layer stack 412 or a direct route to a first join connection (denoted by a circled "+" symbol). The output of spare layer stack 412 is a spare residual 448 having a resolution equal to that of original image 308. Then, as indicated by the circled "+" symbol, in some embodiments, the first join connection sums original image 308 and spare residual 448 on a sample-by-sample basis to generate relaxed original image 418.

[0122] In some embodiments, the preliminary layer stack 412 may include, but is not limited to, any number and / or type of layers trained to map from the source image 308 to the preliminary residual 448. Thus, the input to the preliminary layer stack 412 has the same resolution as the output of the preliminary layer stack 412. As shown, in some embodiments, the preliminary layer stack 412 may include, but is not limited to, convolutional ReLU layer pairs 330(1)-330(K) and a convolutional layer 340(1), where K may be any positive integer. The convolutional ReLU layer pairs 330(1)-330(K) and the convolutional layer 340(1) are as described above in the discussion of FIG. 3 .

[0123] As shown, in some embodiments, the input to downsampling residual block 360 is relaxed original image 418, and the output of downsampling residual block 360 is downsampled image representation 248. The resolution of downsampled image representation 248 is equal to the resolution of relaxed original image 418 divided by downsampling factor 124(2). Thus, the resolution of downsampled image representation 248 is equal to the resolution of original image 308 divided by downsampling factor 124(2). After replacing resampling factor denominator 214(1) with downsampling factor 124(2), the functionality of low-resolution layer stack 362 in trained downsampling CNN 140(2) is similar to the functionality of low-resolution layer stack 362 in trained downsampling CNN 140(1) described above in the description of FIG. 3.

[0124] For ease of explanation, Figure 4 italics the exemplary values ​​associated with the exemplary embodiment of trained downsampling CNN 140(2). In the exemplary embodiment, downsampling factor 124(2) is 2, and source image 308 is a frame of source video 152 having a resolution of 1920x1080. As shown, in the exemplary embodiment, the output of preliminary layer stack 412 is a preliminary residual 448 having a resolution of 1920x1080. Identity residual block 410 then sums source image 308 and preliminary residual 448 on a sample-by-sample basis to produce a relaxed source image 418 having a resolution of 1920x1080.

[0125] In the exemplary embodiment shown in FIG. 4 , downsampling function 380 implements a bicubic downsampling algorithm associated with a scaling factor of “2.” In operation, downsampling function 380 performs the bicubic downsampling algorithm on relaxed original image 418 to generate algorithmically generated downsampled image 388 having a resolution of 960×540. Low-resolution layer set 370 includes, but is not limited to, a convolutional layer associated with an input stride of “2” followed by a ReLU layer. Thus, the output of low-resolution layer stack 362 is downsampled residual 378 having a resolution of 960×540. Downsampling residual block 360 then computes the sum of algorithmically generated downsampled image 388 and downsampled residual 378 on a sample-by-sample basis to generate downsampled image representation 248 having a resolution of 960×540.

[0126] 5 is a flow diagram illustrating method steps for generating a trained down-sampling CNN, according to various embodiments. Note that the method steps are described with reference to the systems illustrated in FIGS. 1-4, but one skilled in the art will understand that the scope of the present embodiments includes any system configured to perform the method steps in any order.

[0127] As shown, the method 500 begins at step 502, in which the configuration engine 210 generates a downsampling CNN 240 based on the downsampling factor 124. The downsampling CNN 240 may include, but is not limited to, two residual blocks. Then, at step 504, the iteration engine 220 generates a number of image patches 228 based on the training image database 122, divides the generated image patches 228 into a number of batches 222, and selects a first one of the batches 222.

[0128] Then, in step 506, for each image patch 228 included in a selected batch 222 from the number of batches 222, the reconstruction error engine 232 uses the downsampling CNN 240 to generate a downsampled image representation 248 of the image patch 228. In step 508, for each image patch 228 included in the selected batch 222, the upsampling engine 250 upsamples the downsampled image representation 248 of the image patch 228 by the downsampling factor 124 to generate a reconstructed image 258 corresponding to the image patch 228.

[0129] Then, in step 510, for each image patch 228 included in the selected batch 222, the reconstruction error engine 232 calculates a reconstruction error 260 based on the image patch 228 and the reconstructed image 258 corresponding to the image patch 228. In step 512, the iteration engine 220 calculates an iteration error 270 based on the multiple reconstruction errors 260 of the selected batch 222, and updates the parameters of the downsampling CNN 240 based on the iteration error 270.

[0130] Then, in step 514, the iteration engine 220 determines whether the selected batch 222 is the last batch 222. If, in step 514, the iteration engine 220 determines that the selected batch 222 is not the last batch 222, the method 500 proceeds to step 516. Then, in step 516, the iteration engine 220 selects the next batch 222 of the number of batches 222, and the method 500 returns to step 506, where the reconstruction error engine 232 uses the downsampling CNN 240 to generate downsampled image representations 248 of the image patches 228 in the newly selected batch 222.

[0131] On the other hand, if, at step 514, the iteration engine 220 determines that the selected batch 222 is the last batch 222, then the method 500 proceeds to step 518. Then, at step 518, the iteration engine 220 determines whether the current epoch is the last epoch. If, at step 518, the iteration engine 220 determines that the current epoch is not the last epoch, then the method 500 proceeds to step 520. Then, at step 520, the iteration engine 220 selects a first batch 222 of the number of batches 222, and the method 500 returns to step 506, where the reconstruction error engine 232 uses the downsampling CNN 240 to generate downsampled image representations 248 of the image patches 228 in the first batch 222.

[0132] However, if, at step 518, iteration engine 220 determines that the current epoch is the last epoch, method 500 proceeds directly to step 522. Then, at step 522, iteration engine 220 saves downsampling CNN 240 as a trained downsampling CNN 140 corresponding to downsampling factor 124, and optionally transmits the trained downsampling CNN 140 to any number of other software applications. Method 500 then ends.

[0133] Figure 6 is a flow diagram illustrating method steps for encoding source video, according to various embodiments. Note that although the method steps are described with reference to the systems shown in Figures 1-4, one skilled in the art will understand that the scope of the present embodiments includes any system configured to perform the method steps in any order.

[0134] As shown, method 600 begins in step 602, in which the downsampling / encoding application 150 obtains (e.g., receives, reads from memory, etc.) trained downsampling CNNs 140(1)-140(N), each associated with a downsampling factor 124(1)-124(N). Then, in step 604, the downsampling / encoding application 150 receives source video 152 for encoding at any number of coding points. In step 606, the downsampling / encoding application 150 determines a subset of downsampling factors 124 based on the resolutions specified at the coding points and the resolution of the source video 152.

[0135] Then, in step 608, the downsampling / encoding application 150 runs the trained downsampling CNN 140 corresponding to each downsampling factor 124 on the original video 152 for each of the multiple downsampling factors 124 included in the subset of downsampling factors 124 to generate a low-resolution video corresponding to each downsampling factor 124.

[0136] Then, in step 610, the downsampling / encoding application 150 encodes the original video 152 or encodes the low-resolution video having the resolution specified at each coding point based on the encoding parameter set specified at that coding point to generate encoded video 148 corresponding to each coding point. Then, in step 612, the downsampling / encoding application 150 stores the encoded video 148 in memory accessible to at least one other software application, or transmits the encoded video 148 to any number of other software applications (e.g., ladder application 160). Method 600 then ends.

[0137] In other words, the techniques of this disclosure can reduce the overall degradation in image quality that often occurs when streaming media titles. In some embodiments, the training application generates a trained downsampling CNN based on a downsampling factor and a training image database. The downsampling factor can be any integer or any non-integer. The training image database can include any number and / or type of training images. The training application includes, but is not limited to, a configuration engine, a reconstruction error engine, and an iteration engine.

[0138] If the downsampling factor is a non-integer, the configuration engine sets the resampling factor numerator and the resampling factor denominator to be equal to the numerator and denominator, respectively, of an irreducible fraction representing the downsampling factor.The configuration engine then generates a first version of the downsampling CNN, comprising (but not limited to), an upsampling residual block followed by a downsampling residual block.

[0139] The upsampling residual block includes, but is not limited to, a trainable high-resolution layer stack in the main path and a differentiable and configurable upsampling function in the shortcut path. The first layer in the trainable upsampling layers is a deconvolutional layer with an output stride equal to the resampling scale numerator. The complementary upsampling function performs upsampling by an upsampling factor equal to the resampling scale numerator.

[0140] The downsampling residual block includes, but is not limited to, a trainable low-resolution layer stack in the main path and a differentiable and configurable downsampling function in the shortcut path. The first layer in the trainable downsampling layer is a convolutional layer with an input stride equal to the resampling scale denominator. The complementary downsampling function downsamples by a factor equal to the resampling scale denominator.

[0141] On the other hand, if the downsampling factor is an integer, the configuration engine generates a first version of the downsampling CNN that includes (but is not limited to) an identity residual block followed by a downsampling residual block. The identity residual block is a degenerate version of the upsampling residual block, including, but not limited to, a trainable spare layer stack in the main path and an identity function in the shortcut path. The configuration engine uses the downsampling factor as the denominator of the resampling factor when configuring the downsampling residual block.

[0142] For each training image in the training image database, the iteration engine generates multiple image patches. The iteration engine then divides these image patches into any number of batches and performs batch iterations for any number of epochs. To perform batch iterations for a given batch, the iteration engine first selects image patches from the batch. The iteration engine then inputs the selected image patches into the latest version of the downsampling CNN. In response, the downsampling CNN outputs downsampled image representations of the selected image patches.

[0143] The iteration engine then performs a differentiable upsampling algorithm on each downsampled image representation. The differentiable upsampling algorithm generates a reconstructed image corresponding to the selected image patch by upsampling each downsampled image representation by a downsampling factor. The iteration engine then establishes a reconstruction error associated with each selected image patch. The reconstruction error is equal to the mean squared error of the reconstructed image corresponding to each selected image patch for that selected image patch. The iteration engine then establishes an iteration error. The iteration error is equal to the average of the reconstruction errors for the selected image patches. The iteration engine then performs any number of backpropagation steps and any number of gradient descent steps, updates the parameters of the downsampling CNN based on the iteration error, and completes the iteration.

[0144] After the iteration engine finishes executing the final epoch, the iteration engine saves the trained downsampling CNN to memory and / or transmits the trained downsampling CNN to any number of other software applications. In some embodiments, the training application is configured to generate multiple trained downsampling CNNs, each corresponding to a different downsampling factor. In the same or other embodiments, any number of trained downsampling CNNs replace conventional downsamplers in any number of video coding pipelines.

[0145] At least one technical advantage of the disclosed technology over the prior art is that the use of a trained downsampling CNN can mitigate the degradation in image quality typically experienced with conventional downsamplers. To this end, a training application trains the downsampling CNN to reduce reconstruction error. This reconstruction error approximates the degradation in image quality of a reconstructed image frame due to a reduction in the resolution of the corresponding source image frame. As a result, implementing the trained downsampling CNN in a video coding pipeline can improve the image quality of the reconstructed image for a given bitrate compared to using a conventional video coding pipeline. Furthermore, to achieve a given target image quality, the video coding pipeline can use fewer bits to encode the source image than would be used in a conventional video coding pipeline. Another technical advantage of the disclosed technology is that any number of the trained downsampling CNNs can downsample the source image, and therefore the source video, by a non-integer downsampling factor, despite limitations on the implementation of pooling and striding in the CNN. These technical advantages result in one or more technical improvements over prior art approaches.

[0146] Section 1 In some embodiments, a computer-implemented method for training a neural network to downsample images in a video encoding pipeline includes: executing a first convolutional neural network, the first convolutional neural network comprising at least two residual blocks and associated with a first downsampling scale factor, on a first original image having a first resolution to generate a first downsampled image; executing an upsampling algorithm on the first downsampled image to generate a first reconstructed image having the first resolution; calculating a first reconstruction error based on the first reconstructed image and the first original image; and updating at least one parameter of the first convolutional neural network based on the first reconstruction error to generate a trained convolutional neural network.

[0147] Section 2 2. The computer-implemented method of claim 1, wherein the first downsampling factor comprises a non-integer, and the method further comprises determining a resampling factor numerator and a resampling factor denominator based on the first downsampling factor; generating an upsampling residual block based on the resampling factor numerator; generating a downsampling residual block based on the resampling factor denominator; and concatenating the downsampling residual block with the upsampling residual block to generate a first convolutional neural network.

[0148] Section 3 3. The computer-implemented method of claim 1 or 2, wherein the upsampling residual block comprises a deconvolutional layer having an output stride equal to the resampling scale numerator and a second upsampling algorithm implementing an upsampling scale equal to the resampling scale numerator.

[0149] Section 4 4. The computer-implemented method of any of claims 1 to 3, wherein the first downsampling factor comprises an integer, and the method further comprises generating a downsampled residual block based on the first downsampling factor; and concatenating the downsampled residual block with an identity residual block to generate a first convolutional neural network.

[0150] Section 5 5. The computer-implemented method of any of clauses 1 to 4, wherein the upsampling algorithm is differentiable.

[0151] Section 6 6. The computer-implemented method of any of claims 1 to 5, further comprising configuring the upsampling algorithm to implement an upsampling factor equal to the first downsampling factor.

[0152] Section 7 7. The computer-implemented method of any of claims 1 to 6, wherein the step of calculating the first reconstruction error includes the step of calculating a mean square error of the first reconstructed image relative to the first original image.

[0153] Section 8 8. The computer-implemented method of any of claims 1 to 7, wherein updating at least one parameter of the first convolutional neural network comprises: calculating an iteration error based on a first reconstruction error and at least a second reconstruction error, where the second reconstruction error is associated with a second original image having a second resolution that is not equal to the first resolution; and performing at least one of a backpropagation operation or a gradient descent operation on the first convolutional neural network based on the iteration error to update the at least one parameter.

[0154] Section 9 9. The computer-implemented method of any of claims 1 to 8, wherein the trained convolutional neural network maps from an original image to downsampled images, each of which has a resolution that is a first downsampling factor lower than the resolution of the corresponding original image.

[0155] Section 10 10. The computer-implemented method of any one of claims 1 to 9, wherein the first original image includes at least a portion of a frame included in the training video.

[0156] Section 11 In some embodiments, one or more non-transitory computer-readable media include instructions that, when executed by one or more processors, train a neural network to downsample an image by causing the one or more processors to: execute a first convolutional neural network, the first convolutional neural network comprising at least two residual blocks and associated with a first downsampling scale factor, on a first original image having a first resolution to generate a first downsampled image; upsample the first downsampled image to generate a first reconstructed image having the first resolution; calculate a first reconstruction error based on the first reconstructed image and the first original image; and perform one or more training operations on the first convolutional neural network based on the first reconstruction error to generate a trained convolutional neural network.

[0157] Section 12 12. The one or more non-transitory computer-readable media of claim 11, wherein the first downsampling factor comprises a non-integer, and the instructions further cause the one or more processors to perform the steps of: determining a resampling factor numerator and a resampling factor denominator based on the first downsampling factor; generating an upsampling residual block based on the resampling factor numerator; generating a downsampling residual block based on the resampling factor denominator; and concatenating the downsampling residual block with the upsampling residual block to generate a first convolutional neural network.

[0158] Section 13 13. The one or more non-transitory computer-readable media of claim 11 or 12, wherein the upsampling residual block comprises a deconvolutional layer having an output stride equal to the resampling scale numerator and an upsampling algorithm that implements an upsampling scale equal to the resampling scale numerator.

[0159] Section 14 14. The one or more non-transitory computer-readable media of any of clauses 11-13, wherein the first downsampling factor comprises an integer, and the instructions further cause the one or more processors to perform the steps of: generating a downsampled residual block based on the first downsampling factor; and concatenating the downsampled residual block with an identity residual block to generate a first convolutional neural network.

[0160] Section 15 15. One or more non-transitory computer-readable media described in any of clauses 11 to 14, wherein the step of upsampling the first downsampled image includes performing a bicubic upsampling algorithm or a bilinear upsampling algorithm on the first downsampled image.

[0161] Section 16 16. One or more non-transitory computer-readable media described in any of clauses 11 to 15, wherein upsampling the first downsampled image includes executing an upsampling algorithm that implements an upsampling factor on the first downsampled image that is equal to the first downsampling factor.

[0162] Section 17 17. The one or more non-transitory computer-readable media of any of paragraphs 11 to 16, wherein the step of calculating the first reconstruction error includes the step of calculating a mean square error of the first reconstructed image relative to the first original image.

[0163] Section 18 18. The one or more non-transitory computer-readable media of any of paragraphs 11 to 17, wherein the step of performing one or more training operations includes performing at least one of a backpropagation operation or a gradient descent operation on the first convolutional neural network based on the first reconstruction error to update at least one parameter of the first convolutional neural network.

[0164] Section 19 19. One or more non-transitory computer-readable media described in any of clauses 11 to 18, wherein the instructions further cause the one or more processors to perform a step of running the trained convolutional neural network on each frame of the original image having the second resolution to generate a downsampled image having a resolution that is lower than the second resolution by a first downsampling factor.

[0165] Section 20 In some embodiments, the convolutional neural network comprises a first residual block that maps from an original image to an upsampled image having a resolution that is higher than the resolution of the original image by a resampling scale factor numerator associated with the downsampling scale factor, and a second residual block concatenated with the first residual block that maps from the upsampled image to a downsampled image having a resolution that is lower than the resolution of the original image by the downsampling scale factor.

[0166] Section 21 In some embodiments, a computer-implemented method for downsampling an image includes running a first trained convolutional neural network, comprising at least two residual blocks and associated with a first downsampling factor, on a first original image having a first resolution to generate a first downsampled image having a second resolution lower than the first resolution.

[0167] Section 22 22. The computer-implemented method of claim 21, wherein the at least two residual blocks include an upsampled residual block associated with the numerator of the resampling fraction and a downsampled residual block associated with the denominator of the resampling fraction.

[0168] Section 23 23. The computer-implemented method of clause 21 or clause 22, wherein the upsampling residual block comprises a deconvolutional layer with an output stride equal to the numerator of the resampling fraction.

[0169] Section 24 24. The computer-implemented method of any of clauses 21 to 23, wherein the upsampling residual block includes an upsampling algorithm that implements an upsampling factor equal to the numerator of the resampling fraction.

[0170] Section 25 25. The computer-implemented method of any of clauses 21 to 24, wherein the first downsampling factor is an integer and the at least two residual blocks comprise an identity residual block and a downsampled residual block.

[0171] Section 26 26. The computer-implemented method of any of clauses 21 to 25, wherein the first original image comprises a frame of the original video and the first downsampled image comprises a frame of the downsampled video.

[0172] Section 27 27. The computer-implemented method of any of claims 21 to 26, further comprising performing one or more encoding processes on the first downsampled image to generate an encoded image.

[0173] Combinations of any element recited in any claim and / or any element described herein, in any combination whatsoever, are intended to be within the scope of the contemplated embodiments and protection.

[0174] While various embodiments have been described, this has been done for illustrative purposes only, and the embodiments presented are not intended to be exhaustive or limiting of the embodiments of the present disclosure. Many modifications and variations will be apparent to those skilled in the art that do not depart from the scope and spirit of the embodiments described herein.

[0175] Aspects of the present disclosure may be embodied as a system, a method, or a computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be collectively referred to herein as a "module," a "system," or a "computer." Also, any hardware and / or software technique, process, function, component, engine, module, or system described in this disclosure may be implemented as a single circuit or set of circuits. Furthermore, aspects of the present disclosure may take the form of a computer program product, or may be embodied as a computer-readable program codec on one or more computer-readable medium(s).

[0176] One or more computer-readable media may be utilized in any combination. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific (but non-exhaustive) examples of computer-readable storage media include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, a flash memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof. In the context of this document, a computer-readable storage medium may be any tangible medium that can store a program for use by or in association with a system, apparatus, or device that executes instructions.

[0177] Aspects of the present disclosure have been described above with reference to flowchart illustrations and / or block diagrams illustrating methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can then be provided to a processor of a programmable data processing apparatus, such as a general-purpose computer or a special-purpose computer, to produce a machine. The instructions, when executed by the processor of the programmable data processing apparatus, such as a computer, cause the implementation of the functions / acts specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor may include, but is not limited to, a general-purpose processor, a special-purpose processor, an application-specific processor, or a field-programmable gate array.

[0178] The flowcharts and block diagrams depicted in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams can be considered to represent a module, segment, or portion of code, including one or more executable instructions for implementing the logical function(s) identified therein. It should also be noted that in some alternative implementations, the functions noted in the blocks can be executed in a different order than that depicted in the figures. For example, two blocks shown as consecutive blocks can in fact be executed substantially simultaneously, or in some cases, the blocks can be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, can be implemented by a dedicated hardware-based system that performs the functions or operations identified therein, or by a combination of dedicated hardware and computer instructions.

[0179] It should be noted that while the foregoing description is directed to embodiments of the present disclosure, other and further embodiments of the present disclosure may be devised without departing from the basic scope thereof, which scope is defined by the following claims. Preferred embodiments of the present invention will be described below in detail. Embodiment 1 1. A computer-implemented method for training a neural network to downsample images in a video encoding pipeline, comprising: executing a first convolutional neural network, the first convolutional neural network comprising at least two residual blocks and associated with a first downsampling factor, on a first original image having a first resolution to generate a first downsampled image; performing an upsampling algorithm on the first downsampled image to generate a first reconstructed image having the first resolution; calculating a first reconstruction error based on the first reconstructed image and the first original image; updating at least one parameter of the first convolutional neural network based on the first reconstruction error to generate a trained convolutional neural network; A computer-implemented method comprising: Embodiment 2 the first downsampling factor is a non-integer; The computer-implemented method comprises: determining a resampling factor numerator and a resampling factor denominator based on the first downsampling factor; generating an upsampled residual block based on the resampling scale numerator; generating a downsampled residual block based on the resampling scale denominator; concatenating the downsampled residual block with the upsampled residual block to generate the first convolutional neural network; 2. The computer-implemented method of claim 1, further comprising: Embodiment 3 3. The computer-implemented method of embodiment 2, wherein the upsampling residual block comprises a deconvolutional layer having an output stride equal to the resampling scale numerator, and a second upsampling algorithm implementing an upsampling scale equal to the resampling scale numerator. Embodiment 4 the first downsampling factor is composed of integers; The computer-implemented method comprises: generating a downsampled residual block based on the first downsampling factor; concatenating the downsampled residual block with an identity residual block to generate the first convolutional neural network; 2. The computer-implemented method of claim 1, further comprising: Embodiment 5 2. The computer-implemented method of embodiment 1, wherein the upsampling algorithm is differentiable. Embodiment 6 2. The computer-implemented method of embodiment 1, further comprising configuring the upsampling algorithm to implement an upsampling factor equal to the first downsampling factor. Embodiment 7 2. The computer-implemented method of embodiment 1, wherein calculating the first reconstruction error comprises calculating a mean square error of the first reconstructed image relative to the first original image. Embodiment 8 updating at least one parameter of the first convolutional neural network; calculating an iteration error based on the first reconstruction error and at least a second reconstruction error, the second reconstruction error being associated with a second original image having a second resolution not equal to the first resolution; performing at least one of a backpropagation operation or a gradient descent operation on the first convolutional neural network based on the iteration error to update the at least one parameter; 2. The computer-implemented method of claim 1, comprising: Embodiment 9 the trained convolutional neural network performs a mapping from an original image to a downsampled image; 2. The computer-implemented method of embodiment 1, wherein each of the downsampled images has a resolution that is lower than the resolution of the corresponding original image by the first downsampling factor. Embodiment 10 2. The computer-implemented method of claim 1, wherein the first original image includes at least a portion of a frame included in a training video. Embodiment 11 1. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors, train a neural network to downsample an image, the instructions causing the one or more processors to: executing a first convolutional neural network, the first convolutional neural network comprising at least two residual blocks and associated with a first downsampling factor, on a first original image having a first resolution to generate a first downsampled image; upsampling the first downsampled image to generate a first reconstructed image having the first resolution; calculating a first reconstruction error based on the first reconstructed image and the first original image; performing one or more training processes on the first convolutional neural network based on the first reconstruction error to generate a trained convolutional neural network; and one or more non-transitory computer-readable media for performing said training by causing said medium to execute: Embodiment 12 the first downsampling factor is a non-integer; The instructions to the one or more processors: determining a resampling factor numerator and a resampling factor denominator based on the first downsampling factor; generating an upsampled residual block based on the resampling scale numerator; generating a downsampled residual block based on the resampling scale denominator; concatenating the downsampled residual block with the upsampled residual block to generate the first convolutional neural network; 12. One or more non-transitory computer-readable media as described in embodiment 11, further causing the computer to perform the following steps: Embodiment 13 13. One or more non-transitory computer-readable media as described in embodiment 12, wherein the upsampling residual block comprises a deconvolution layer having an output stride equal to the resampling factor numerator and an upsampling algorithm that implements an upsampling factor equal to the resampling factor numerator. Embodiment 14 the first downsampling factor is composed of integers; The instructions to the one or more processors: generating a downsampled residual block based on the first downsampling factor; concatenating the downsampled residual block with an identity residual block to generate the first convolutional neural network; 12. One or more non-transitory computer-readable media as described in embodiment 11, further causing the computer to perform the following steps: Embodiment 15 One or more non-transitory computer-readable media as described in embodiment 11, wherein the step of upsampling the first downsampled image includes performing a bicubic upsampling algorithm or a bilinear upsampling algorithm on the first downsampled image. Embodiment 16 One or more non-transitory computer-readable media as described in embodiment 11, wherein the step of upsampling the first downsampled image includes a step of executing an upsampling algorithm that implements an upsampling factor on the first downsampled image that is equal to the first downsampling factor. Embodiment 17 12. The one or more non-transitory computer-readable media of embodiment 11, wherein the step of calculating the first reconstruction error includes a step of calculating a mean square error of the first reconstructed image relative to the first original image. Embodiment 18 12. The one or more non-transitory computer-readable media of embodiment 11, wherein the step of performing the one or more training operations includes performing at least one of a backpropagation operation or a gradient descent operation on the first convolutional neural network based on the first reconstruction error to update at least one parameter of the first convolutional neural network. Embodiment 19 The instructions to the one or more processors: Executing the trained convolutional neural network on each frame of an original image having a second resolution to generate a downsampled image having a resolution that is lower than the second resolution by the first downsampling factor. 12. One or more non-transitory computer-readable media as described in embodiment 11, further causing the computer to perform the following steps: Embodiment 20 a first residual block that maps an original image to an upsampled image having a resolution higher than that of the original image by a resampling factor numerator associated with a downsampling factor; a second residual block concatenated with the first residual block, the second residual block performing a mapping from the upsampled image to a downsampled image having a resolution that is lower than the resolution of the original image by the downsampling factor; A convolutional neural network comprising: Embodiment 21 1. A computer-implemented method for downsampling an image, comprising: running a first trained convolutional neural network, comprising at least two residual blocks and associated with a first downsampling factor, on a first original image having a first resolution to generate a first downsampled image having a second resolution lower than the first resolution; A computer-implemented method comprising: Embodiment 22 22. The computer-implemented method of claim 21, wherein the at least two residual blocks include an upsampled residual block associated with the numerator of a resampling fraction and a downsampled residual block associated with the denominator of the resampling fraction. Embodiment 23 23. The computer-implemented method of embodiment 22, wherein the upsampling residual block comprises a deconvolutional layer having an output stride equal to the numerator of the resampling fraction. Embodiment 24 23. The computer-implemented method of embodiment 22, wherein the upsampling residual block includes an upsampling algorithm that implements an upsampling factor equal to the numerator of the resampling fraction. Embodiment 25 the first downsampling factor is composed of integers; 22. The computer-implemented method of embodiment 21, wherein the at least two residual blocks comprise an identity residual block and a downsampled residual block. Embodiment 26 the first original image includes a frame of an original video; 22. The computer-implemented method of embodiment 21, wherein the first downsampled image comprises a frame of a downsampled video. Embodiment 27 22. The computer-implemented method of embodiment 21, further comprising performing one or more encoding operations on the first downsampled image to generate an encoded image. [Explanation of symbols]

[0180] 100 systems 102 training images 110 compute instances 112 processors 116 memory 122 Training Image Database 124 downsampling factor 130 Training Applications 140 pre-trained downsampling CNNs 148 coded video 150 Downsampling / Encoding Applications 152 Original footage 160 Ladder Applications 162 Bitrate Ladder 164 coded video subset 170 Content Delivery Networks (CDNs) 172 Encoded Video Chunk 180 client devices 182 Display Devices 190 Endpoint Applications 192 Reconstructed Video Chunks 202 Configurable non-integer scaling factor CNN 204 configurable integer multiplier CNN 210 Configuration Engine 212 Resampling factor numerator 214 Resampling factor denominator 220 Iteration Engine 222 batch 228 image patches 230 Batch Iterations 232 Reconstruction Error Engine 240 Downsampling CNN 248 Downsampled Image Representation 250 Upsampling Engine 258 reconstructed images 260 Reconstruction error 270 Iteration Error 280 Parameter update process 308 original images 310 Upsampling Residual Block 312 high resolution layer stack 318 upsampled images 320 high resolution layer set 330 Convolutional ReLU Layer Pairs 340 convolutional layers 348 Upsampling Residual 350 Upsampling Functions 358 Algorithmically Generated Upsampled Images 360 Downsampled Residual Block 362 low-res layer stack 370 low-resolution layer set 378 Downsampling Residuals 380 Downsampling Functions 388 Algorithmically Generated Downsampled Images 410 Identity Residual Block 412 spare layer stack 418 Original image with mitigation processing 448 Preliminary Residuals

Claims

1. 1. A computer-implemented method for training a neural network to downsample images in a video encoding pipeline, comprising: executing a first convolutional neural network, the first convolutional neural network comprising at least two residual blocks and associated with a first downsampling factor, on a first original image having a first resolution to generate a first downsampled image; performing an upsampling algorithm on the first downsampled image to generate a first reconstructed image having the first resolution; calculating a first reconstruction error based on the first reconstructed image and the first original image; updating at least one parameter of the first convolutional neural network based on the first reconstruction error to generate a trained convolutional neural network; Including, the residual block maps the input to the residual block to a residual and adds the resulting residual to a function of the input to the residual block to generate an output of the residual block; A computer-implemented method, wherein each of the downsampled images has a resolution that is less than the resolution of the corresponding original image by the first downsampling factor.

2. the first downsampling factor is a non-integer; The computer-implemented method comprises: determining a resampling factor numerator and a resampling factor denominator based on the first downsampling factor; generating an upsampled residual block based on the resampling scale numerator; generating a downsampled residual block based on the resampling scale denominator; concatenating the downsampled residual block with the upsampled residual block to generate the first convolutional neural network; The computer-implemented method of claim 1 further comprising:

3. 3. The computer-implemented method of claim 2, wherein the upsampling residual block comprises a deconvolutional layer having an output stride equal to the resampling scale numerator and a second upsampling algorithm implementing an upsampling scale equal to the resampling scale numerator.

4. the first downsampling factor is composed of integers; The computer-implemented method comprises: generating a downsampled residual block based on the first downsampling factor; concatenating the downsampled residual block with an identity residual block to generate the first convolutional neural network; The computer-implemented method of claim 1 further comprising:

5. The computer-implemented method of claim 1 , wherein the upsampling algorithm is differentiable.

6. The computer-implemented method of claim 1 , further comprising configuring the upsampling algorithm to implement an upsampling factor equal to the first downsampling factor.

7. 2. The computer-implemented method of claim 1, wherein calculating the first reconstruction error comprises calculating a mean square error of the first reconstructed image relative to the first original image.

8. updating at least one parameter of the first convolutional neural network; calculating an iteration error based on the first reconstruction error and at least a second reconstruction error, the second reconstruction error being associated with a second original image having a second resolution not equal to the first resolution; performing at least one of a backpropagation operation or a gradient descent operation on the first convolutional neural network based on the iteration error to update the at least one parameter; 10. The computer-implemented method of claim 1, comprising:

9. 10. The computer-implemented method of claim 1, wherein the trained convolutional neural network maps from an original image to a downsampled image.

10. 1. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors, train a neural network to downsample an image, the instructions causing the one or more processors to: executing a first convolutional neural network, the first convolutional neural network comprising at least two residual blocks and associated with a first downsampling factor, on a first original image having a first resolution to generate a first downsampled image; upsampling the first downsampled image to generate a first reconstructed image having the first resolution; calculating a first reconstruction error based on the first reconstructed image and the first original image; performing one or more training processes on the first convolutional neural network based on the first reconstruction error to generate a trained convolutional neural network; Execute performing said training by mapping an input to said residual block to a residual and adding the resulting residual to a function of the input to said residual block to generate an output of said residual block; One or more non-transitory computer-readable media, wherein each of the downsampled images has a resolution that is lower than a resolution of the corresponding original image by the first downsampling factor.

11. 1. A computer-implemented method for downsampling an image, comprising: running a first trained convolutional neural network, comprising at least two residual blocks and associated with a first downsampling factor, on a first original image having a first resolution to generate a first downsampled image having a second resolution lower than the first resolution; Including, the residual block maps the input to the residual block to a residual and adds the resulting residual to a function of the input to the residual block to generate an output of the residual block; A computer-implemented method, wherein each of the downsampled images has a resolution that is less than the resolution of the corresponding original image by the first downsampling factor.

12. 12. The computer-implemented method of claim 11, wherein the at least two residual blocks comprise an upsampled residual block associated with a numerator of a resampling fraction and a downsampled residual block associated with a denominator of the resampling fraction.

13. 13. The computer-implemented method of claim 12, wherein the upsampling residual block comprises a deconvolutional layer having an output stride equal to the numerator of the resampling fraction.

14. The computer-implemented method of claim 12 , wherein the upsampling residual block includes an upsampling algorithm that implements an upsampling factor equal to the numerator of the resampling fraction.

15. the first downsampling factor is composed of integers; The computer-implemented method of claim 11 , wherein the at least two residual blocks comprise an identity residual block and a downsampled residual block.

16. the first original image includes a frame of an original video; The computer-implemented method of claim 11 , wherein the first downsampled image comprises a frame of a downsampled video.

17. The computer-implemented method of claim 11 , further comprising performing one or more encoding operations on the first downsampled image to generate an encoded image.

Images