Selectively enhancing compressed digital content

Abstract

The present disclosure relates to systems, methods, and computer-readable media for selectively enhancing digital image and video content. For example, a system disclosed herein may encode raw video content to compress and decompress the raw video content. The system described herein may further identify information of the region of interest for identifying portions of the decompressed video content to analyze and remove one or more compression artifacts found therein. The systems described herein may further enhance the decompressed video content by increasing display resolution. By identifying the region of interest and selectively enhancing the digital video content, the system described herein can reduce the consumption of bandwidth and processing resources while maintaining high visual quality of the digital content.

Description

[0001] This invention patent application is a divisional application of the invention patent application with international application number PCT / US2020 / 030330, international application date April 29, 2020, application number 202080042707.2 entering the Chinese national phase, and name “Selectively Enhancing Compressed Digital Content”. Background Art

[0002] In recent years, the use of computing devices (e.g., mobile devices, personal computers) to receive, store, edit, stream, or otherwise display digital media has increased dramatically. In fact, it is now common for consumer electronic devices to display real-time video from multiple different remote sources. In addition, as personal computers, mobile devices, game consoles, and other consumer electronic devices become more popular, a wider variety of electronic devices with different computing and display capabilities are being used in various digital media applications. In addition, as computing devices become more complex, the display capabilities of computing devices have also been similarly improved.

[0003] Although computing devices provide useful tools in displaying digital media, conventional systems for transmitting and displaying digital media include various problems and shortcomings. For example, as the quality of digital media increases, connection resources (e.g., bandwidth) are often insufficient to enable content providers to stream or otherwise provide high-quality media. In addition, even in situations where connection resources are sufficient and can be used to provide high-quality media content, consumers are often frustrated by the various costs associated with downloading and processing large amounts of digital content. In addition, conventional encryption and decryption techniques used to generate and transmit media content often result in low-quality content displays.

[0004] These and other problems exist in delivering and displaying digital video content. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 An example environment including an encoder system and a video enhancement system in accordance with one or more embodiments is illustrated.

[0006] Figure 2 Example implementations of encoder systems and video enhancement systems in accordance with one or more embodiments are illustrated.

[0007] Figures 3A-3C Further example implementations of encoder systems and video enhancement systems in accordance with one or more embodiments are illustrated.

[0008] Figure 4

[0013] Example implementations involving identifying regions of interest of video frames in accordance with one or more embodiments are illustrated.

[0009] Figure 5

[0013] Example methods for enhancing digital video content in accordance with one or more embodiments are illustrated.

[0010] Figure 6 Example methods for encoding video content and identifying regions of interest for enhancing digital video content in accordance with one or more embodiments are illustrated.

[0011] Figure 7 Some components that may be included within a computer system are illustrated. DETAILED DESCRIPTION

[0012] The present disclosure generally relates to encoder systems and digital video enhancement systems for selectively enhancing video content based on identified important areas within the video content. Specifically, as will be discussed in further detail below, a server device may include an encoder system that compresses video content to be streamed (e.g., live streaming) or otherwise provided to a client device. The encoder system can generate a compressed digital video by applying a compression or encoding algorithm to the video content before the compressed video content is transmitted to the client device. In addition, as will be discussed in further detail below, the encoder system can identify information of an area of ​​interest and provide it to a client device associated with an important portion of the video content (e.g., pixel area and / or color channel) to enable the client device to selectively repair, refine, and otherwise enhance the portions of the video content associated with the information of the area of ​​interest.

[0013] In addition to the encoder system, the present disclosure also relates to a video enhancement system for selectively repairing, improving and otherwise enhancing selected portions of digital video content. For example, as will be discussed in further detail below, the video enhancement system may identify a region of interest of a decoded video frame (e.g., a decompressed video frame) corresponding to one or more important portions of a video frame. The video enhancement system may then apply a denoising model to the decoded video frame to remove one or more compression artifacts introduced into the digital content during the compression and decompression process. Once the decoded video frame is denoised, the video enhancement system may further refine the decoded video frame by interpolating pixels, upsampling, or otherwise increasing the pixel resolution of the repaired video frame before displaying the output video frame via a graphical user interface of a display device. In fact, the encoder system and the video enhancement system may apply a similar process to any number of video frames of a decompressed or streaming digital video.

[0014] As will be discussed in further detail herein, the present disclosure includes multiple practical applications that provide benefits and / or solve problems associated with delivering and displaying digital video content via a graphical user interface of a client device. For example, by identifying regions of interest in a video frame, a video enhancement system can significantly reduce processing resources consumed in repairing or otherwise enhancing the digital video content. In fact, by selectively repairing artifacts in important portions of a video frame, a video enhancement system can generate quality video frames in which the most important portions of the video frame are preserved and have high quality even in situations where the client device has limited processing resources or where consuming more processing resources results in video delays and / or poor battery performance.

[0015] In addition, by identifying regions of interest, the video enhancement system can utilize one or more robust and accurate machine learning models while preserving the overhead of computing resources. For example, the video enhancement system can selectively apply a denoising model trained to remove artifacts from an input image to an identified region of interest of a decoded video frame to remove artifacts from the identified region of interest, rather than analyzing every pixel or additional portion of the decoded video frame. As a further example, the video enhancement system can apply a super-resolution model to a repaired image (e.g., a repaired decoded video frame) or select portions of a repaired image to further enhance the image while efficiently utilizing computing resources. In addition, one or more embodiments described herein relate to a machine learning model that is trained to generate a segmentation map and / or image buffer to indicate key regions of interest in different implementations.

[0016] As will be discussed in further detail below in conjunction with a number of embodiments, the video enhancement system and encoder system can collaboratively identify and utilize information about regions of interest in a variety of ways. For example, where a developer of video content (e.g., a game developer) is aware of cloud computing or compression considerations, the encoder system and / or video enhancement system can utilize information about regions of interest in an effective and efficient manner when improving video quality. For example, a game developer can cause a video generation system to provide information about regions of interest in conjunction with provided game content. In addition, even in the case where the original content source does not generate or provide information about regions of interest, the examples described herein also relate to various ways in which information about regions of interest is identified by an encoder system and / or video enhancement system to more effectively enhance digital video content.

[0017] By more efficiently utilizing processing and bandwidth resources, the systems described herein can provide significant flexibility in generating and delivering digital video content. For example, by more efficiently utilizing processing and bandwidth resources, the systems described herein facilitate higher quality output without increasing bandwidth consumption. Additionally or alternatively, the systems described herein can maintain a high or threshold quality of video content while reducing bandwidth consumption. In fact, depending on any number of user or system preferences and settings (e.g., bandwidth budget, cellular data usage restrictions, selected graphics settings, battery consumption settings), the systems described herein can provide many different benefits in the operation of client devices when displaying digital video content.

[0018] As illustrated in the foregoing discussion, the present disclosure utilizes various terms to describe the features and advantages of the systems described herein. Additional details about the meaning of such terms are now provided. For example, as used herein, "digital video", "digital video content" or "video content" may interchangeably refer to digital data representing a sequence of digital images (e.g., multiple digital video frames). Digital video may include digital data or digital files that can be displayed via a graphical user interface of a display of a computing device. For example, a digital video may include an image sequence that includes pixel data representing discrete units or elements of a digital image that, when combined, form displayable content of a digital image. A digital video may have a corresponding frame rate, resolution, or duration. As an example, a digital video may refer to a digital file having one of the following extensions: AVI, FLV, WMV, MOV, MP4. As used herein, a "video frame" or "digital video frame" refers to a discrete frame of video content from a digital video content.

[0019] In one or more implementations described herein, digital video may refer to compressed digital video. As used herein, "compressed digital video" or "encoded video" refers to a digital video whose pixel data has been reduced using one of a variety of compression techniques. For example, a compressed digital video may refer to a digital video (or a series of video frames) that has been compressed using a lossy or lossless compression algorithm. As a further example, in one or more embodiments, the compressed digital video is compressed using one or more block-oriented motion-compensated video compression standards. For example, the compressed digital video may be compressed using a format including H264, H.265, MPEG-4, VP9, ​​VP10, or any other encoding or compression format.

[0020] In contrast to compressed digital video, in one or more embodiments described herein, digital video may refer to decompressed digital video. As used herein, "decompressed digital video" or "decoded digital video" refers to digital video that has been generated from a compressed digital video to approximate the original form of the digital video. For example, in one or more implementations described herein, the decompressed digital video is generated by applying a decoding or decompression algorithm to a compressed digital video file (or compressed digital video content stream) based on the format of the compressed digital video and / or the compression algorithm applied to the original video to generate the compressed digital video. In one or more embodiments described herein, the decompressed digital video refers to the decompressed portion of the streaming video content that is decompressed when received and presented via a graphical user interface.

[0021] In one or more embodiments described herein, a region of interest is identified for one or more video frames. As used herein, a "region of interest" refers to a discrete portion of a video frame (or multiple video frames). For example, a region of interest may refer to a cropped portion of an image or a pixel region from a digital video frame. In one or more implementations, a region of interest refers to a discrete region of multiple pixels at different locations within a digital image. For example, a region of interest may refer to a first pixel region and a second non-overlapping pixel region from a corresponding video frame. A region of interest may be consistent across multiple video frames, or may vary frame by frame. In one or more embodiments, a region of interest refers to a channel (e.g., a color channel, a chroma channel, a luminance channel) that forms a portion of an image, which forms a digital image when the channel is combined with an additional channel. More examples of regions of interest will be discussed in more detail below.

[0022] In one or more embodiments described herein, various systems utilize one or more machine learning models to perform various functions with respect to digital video content. As used herein, "machine learning model" refers to one or more computer algorithms or models (e.g., classification models, regression models) that can be tuned (e.g., trained) to approximate unknown functions based on training inputs. For example, a machine learning model can refer to a neural network (e.g., a convolutional neural network, a recurrent neural network) or other architecture that learns and approximates complex functions and generates outputs based on one or more inputs provided to the machine learning model. As will be discussed in conjunction with various examples herein, different machine learning models can generate different types of outputs (e.g., inpainted images, upsampled images, segmentation masks) based on how the machine learning model is trained to generate these outputs.

[0023] Additional details about the system for delivering and enhancing digital video content will now be provided with reference to illustrative figures depicting example implementations. For example, Figure 1An example environment 100 is illustrated in which digital video is delivered to a client device and selectively enhanced according to one or more embodiments described herein. Figure 1 As shown, environment 100 includes one or more server devices 102 and client devices 104. Server device 102 includes a video generation system 106 and an encoder system 108. As further shown, client device 104 includes a decoder system 110 and a video enhancement system 112.

[0024] like Figure 1 As further shown, the server device 102 and the client device 104 can communicate with each other directly or indirectly via a network 114. The network 114 can include one or more networks and can use one or more communication platforms or technologies suitable for transmitting data. The network 114 can refer to any data link capable of transmitting electronic data between devices and / or modules of the environment 100. The network 114 can refer to a hardwired network, a wireless network, or a combination of hardwired and wireless networks. In one or more embodiments, the network 114 includes the Internet.

[0025] The client device 104 may refer to various types of computing devices. For example, the client device 104 may include a mobile device, such as a mobile phone, a smart phone, a personal digital assistant (PDA), a tablet, or a laptop computer. Additionally or alternatively, the client device 104 may include a non-mobile device, such as a desktop computer, a server device, or other non-portable device. In one or more embodiments, the client device refers to a gaming device, such as a dedicated handheld gaming device or a video game console that communicates with a display device. In one or more implementations, the client device 104 refers to a system of multiple devices (e.g., a game console and an associated display device). The client device 104 may additionally include a graphical user interface thereon (e.g., a screen of a mobile device). Additionally or alternatively, the client device 104 may be communicatively coupled (e.g., wired or wirelessly) to a display device having a graphical user interface thereon for providing a display of digital video content. The server device 102 may similarly refer to various types of computing devices. Each device of the environment 100 may include the following in combination: Figure 7 Describes features and functionality.

[0026] Additional details will now be presented in conjunction with an example implementation in which components of environment 100 facilitate efficient delivery and enhancement of digital video for display via a graphical user interface of client device 104 . Figure 2 An example implementation is illustrated in which the video enhancement system 112 includes a denoising system 202 and a super-resolution system 204 for enhancing digital video content and providing for display of the enhanced digital video content via a graphical user interface of a display device 222 .

[0027] like Figure 2 As shown, the video generation system 106 can generate raw video content 208. The raw video content can refer to live video generated by the video generation system 106 on the same or different server device as the encoder system 108. The raw video content 208 can refer to video game content generated by a game server for real-time streaming to the client device 104. The raw video content 208 can have a native resolution and bit rate, depending on the capabilities and configuration of the video generation system 106 and / or the development specifications of the client device 104, the video generation server, or other video generation application.

[0028] In one or more embodiments, video generation system 106 generates information about the region of interest and provides it to one or more components of environment 100. For example, in Figure 2 In the illustrated example, the video generation system 106 generates and provides information about the region of interest to the region of interest manager 206. This region of interest information may include information provided by the developer of the original video content 208. For example, the region of interest information may include an indication of an input element, a head-up display (HUD) element (e.g., an element displayed as part of a user interface), or any other graphical user interface element to be displayed with the digital video content. The region of interest manager 206 may be implemented on the server device 102, the client device 104, or include different features implemented between different devices. Additional details thereof related to identifying and utilizing the region of interest information will be discussed in further detail below.

[0029] In providing raw video content 208 to the client device 104, the encoder system 108 may be used to compress the raw video content 208 and generate a compressed digital video 210. Specifically, the encoder system 108 may generate the compressed digital video 210 in a format that can be transmitted to the client device 104 over the network 114. For example, the encoder system 108 may apply an encoding algorithm to compress the raw video content 208 and provide the compressed video content to the client device 104 via a wireless connection. In one or more embodiments described herein, the compressed video 210 is streamed to the client device 104 in real time.

[0030] like Figure 1As shown, encoder system 108 may refer to a single instance of encoder system 108 that includes features and functionality associated with various examples described herein that corresponds to an associated video content generator 106. Alternatively, in one or more embodiments, server device 102 may include multiple video content generators 106, each of which provides video game content to a shared encoder system 108 that is equipped to encode video content and provide compressed digital video to any number of devices. Thus, while one or more embodiments of encoder system 108 involve implementing encoder system 108 as part of the same hardware and / or software of a video content generator, encoder system 108 may alternatively be implemented on a separate hardware device that is independent of the video content generator implementation.

[0031] After receiving the compressed video content, the client device 104 may apply the decoder 110 to the compressed video 210 to generate a decompressed digital video 212 including a plurality of decompressed video frames. Specifically, the decoder 110 may apply a decoding algorithm (e.g., depending on the format of the compressed video 210) to the compressed video 210 to reconstruct the digital video content from the encoded data to approximate the original video generated by the video generation system 106. The decompressed video 212 may include a bit rate and / or resolution corresponding to the bit rate and resolution of the original video content 208 provided by the video generation system 106. Alternatively, in one or more embodiments, the decompressed video 212 may include a lower resolution and / or a lower bit rate than the original video content 208. For example, in the case where the network connection between the client device 104 and the server device is limited, the decompressed video 212 may have a lower resolution or a lower bit rate than the resolution and bit rate of the original video content 208 to preserve bandwidth or other network resources.

[0032] like Figure 2 As shown, as a result of compressing and decompressing the original video content 208, one or more video frames may include one or more decompression artifacts 214. As used herein, "artifacts" or "decompression artifacts" refer to any data loss between the original video content 208 and the decompressed digital video content 212 due to compressing and decompressing the digital video. For example, compression artifacts may refer to defects such as lost pixels, pixelation effects, downsampling due to encoding and decoding, blurriness, content distortion, tinting, noise, graininess, or any other effect caused by data loss due to compressing and decompressing digital content. Compression artifacts may include localized artifacts (e.g., such as Figure 2Alternatively, compression artifacts may refer to the effect of lost data on the entire image (eg, graininess or reduced resolution across the entire image).

[0033] In one or more embodiments, the decompressed video content 212 is provided to the region of interest manager 206. The region of interest manager 206 can identify or otherwise determine the region of interest within the video content in a variety of ways. In one or more embodiments, the region of interest manager 206 determines a region of pixels within one or more video frames based on the region of interest information received from the video generation system 106. For example, the region of interest manager 206 may receive an identification of a head-up display (HUD) element within the video content and map the head-up display (HUD) element to a region of pixels within the received decompressed video content 212. The region of interest manager 206 can use the region of interest information to generate a segmentation mask 216 or other image buffer that includes an identified region of pixels corresponding to the region of interest information. Figure 3A-4 Further information related to information identifying regions of interest is discussed in more detail.

[0034] In addition to the segmentation mask 216, the decompressed digital video content 212 can be provided to the denoising system 202. The denoising system 202 can include a denoising model (e.g., a machine learning model) that is trained to extract, repair, or otherwise remove decompression artifacts included within the decompressed video content 212. For example, the denoising system 202 can apply the machine learning model to a video frame including the artifacts 214 and generate an output image that approximates the original video content 208 based on the training data previously used in training the denoising model.

[0035] The denoising system 202 can be trained to remove compression artifacts in a variety of ways. For example, in one or more embodiments, the denoising system 202 can include a machine learning model trained by providing image pairs including a first plurality of original images before compression and a second plurality of digital images after compression and decompression. In this way, the denoising system 202 can be trained to approximate or estimate how digital images with various different compression artifacts will appear before the compression artifacts are introduced via the compression and decompression process.

[0036] Although the denoising system 202 can be trained to remove artifacts in general, in one or more embodiments, the denoising system 202 is trained or refined based on a specific type or source of video content. As an example, in some examples, the original video content 208 can refer to a game or other application where the region of interest is uniform or limited to a limited set of regions within the corresponding video frame. Where the region of interest is known, the denoising system 202 can be specifically trained to remove artifacts in a specific region of interest. In this way, information about the region of interest can be used to more effectively train the denoising system 202 for one or more specific applications.

[0037] like Figure 2 As shown, the denoising system 202 can provide the repaired video frame 218 as an input to the super-resolution system 204 to further enhance the video content. For example, the super-resolution system 204 may include a super-resolution model (e.g., a machine learning model) trained to upsample or otherwise increase the pixel resolution of one or more video frames. Thus, the super-resolution system 204 can receive the repaired video frame 218 and generate a high-resolution video frame 220 for display via a display device 222. As used herein, a "high-resolution video frame" may refer to a video frame having a higher resolution than a decompressed video frame generated by the decoder 110.

[0038] In one or more embodiments, the super-resolution system 204 includes a machine learning model trained to generate high-resolution video frames. Specifically, the super-resolution system 204 can be trained based on a pair of high-resolution images and corresponding low-resolution images. In one or more implementations, the image pair refers to an image that does not include artifacts (e.g., refers to a different set of training images used to train a denoising model). For example, although in Figure 1 2, but the server device may include a model training system that generates or otherwise obtains a downsampled version of the original video content and trains the super-resolution system 204 (or a model implemented by the super-resolution system 204) based on pairs of video frames from the original video content and corresponding downsampled versions of the video frames from the original video content. In this way, the super-resolution system 204 can be trained to approximate an upsampled version of the digital video content based on the training image pairs.

[0039] like Figure 2As shown, the high-resolution video frame 220 can be displayed via a graphical user interface. For example, the high-resolution video frame 220 can be streamed to a display device (e.g., a mobile device) for display while the high-resolution video frame 220 is being generated. Note that the process of denoising and increasing the resolution (e.g., upsampling, interpolating pixels) of the digital video frame can be performed locally on the client device 104. In this way, the video enhancement system 112 can repair and generate the high-resolution video frame 220 without consuming additional connection resources (e.g., bandwidth, cellular data).

[0040] Figure 2 Each of the components shown in the can perform one or more of the above processes according to various settings or preferences to achieve a variety of different benefits. For example, in one or more embodiments, the video enhancement system 112 can denoise (e.g., remove artifacts) the decompressed video frames 212 and generate high-resolution output video frames 220 to approximate video frames having a resolution and bit rate similar to the original video content 208. For example, in order to save bandwidth resources, the encoder system 108 can compress the digital video to generate encoded video content 210 with a lower resolution or bit rate than the original video content 208 to reduce bandwidth resources consumed when providing the video content over the network 114. Alternatively, the encoder system 108 still generates a lossless or near-lossless compression of the original video content 208, while still generating a very high quality output image by further upsampling or otherwise enhancing the resolution after removing one or more compression artifacts.

[0041] In one or more embodiments, the video enhancement system 112 may omit the super-resolution system 104 or simply provide the repaired video frames 218 as output to be displayed via the display device 222. For example, in the event that the video enhancement system 112 does not include the super-resolution system 204 or the video quality has reached a sufficient level of quality to be displayed, the video enhancement system 112 may simply provide the repaired video frames 218 to the display device 222 for display via a graphical user interface.

[0042] In yet another example, one or both of the components 202-204 of the video enhancement system 112 may be applied to selected portions of the digital image provided to the video enhancement system 112. For example, as mentioned above, the denoising system 202 may apply a denoising model specifically to portions of the video frame corresponding to the identified region of interest, without applying the denoising system 202 to other portions of the video frame. Similarly, the super-resolution system 204 may apply a super-resolution model to selected portions of the video frame to selectively upsample portions of the video frame so that those portions of the video frame identified as important have higher quality visual content. In one or more implementations, selectively upsampling sub-regions of the video frame may involve applying a machine learning-based algorithm to the key regions while applying a simpler upsampling function to other regions of the video frame so that the video frame is upsampled to the same target resolution.

[0043] As an alternative to selectively applying various models to exclusive portions of video frames, the video enhancement system 112 may further prioritize processing of video frames based on identified regions of interest. For example, in one or more embodiments, the video enhancement system 112 may prioritize denoising and / or improving resolution of selected portions of video frames for identified regions of interest while still denoising and / or improving resolution of other portions of the video frames depending on the processing and bandwidth budget. For example, where settings or preferences indicate a maximum computational budget or bandwidth budget associated with a minimum or preferred output quality, the video enhancement system 112 may prioritize enhancing (e.g., denoising, improving resolution) video frames to the identified regions of interest while spending any additional available resources to enhance other portions of the video frames as the computational and / or bandwidth budget permits. In addition, in one or more embodiments, the video enhancement system 112 may simply apply more stringent or robust enhancements (e.g., more robust denoising) to the domain of interest while applying less stringent or robust enhancements to other regions of the video frame, rather than exclusively enhancing portions or prioritizing enhancements.

[0044] As mentioned above, the system described herein can identify and utilize information about regions of interest in a variety of ways. For example, Figures 3A-3C Example implementations are illustrated in which the region of interest manager 206 may be implemented on a server device 102 or a client device 104. It should be appreciated that features and functions described in conjunction with one or more specific illustrated examples may be similarly applied to other examples described herein.

[0045] Figure 3A An example implementation is illustrated in which the video enhancement system 112 receives information about the region of interest from a remote source. Figure 3AAs shown, the denoising system 202 may receive (e.g., from the decoder 110) a decompressed video frame 302 that includes digital content reconstructed from the compressed digital video provided from the encoder system 108. For example, the decompressed video frame 302 may include a video frame from video game content generated by the video generation system 106.

[0046] like Figure 3A As further shown, the decompressed video frame 302 includes a plurality of compression artifacts 304a-c. For example, the video frame 302 includes a first compression artifact 304a at the upper left portion of the video frame 302, a second compression artifact 304b at the middle right portion of the video frame 302, and a third compression artifact 304c at the bottom middle portion of the video frame 302. The compression artifacts 304a-b may represent any of a variety of types of localized defects or inaccuracies within the video frame due to compression and decompression of video content. As further shown, the example video frame 302 is from a racing video game application that includes display content such as a speed indicator and a level indicator.

[0047] In addition to receiving the decompressed video frame 302, the denoising system 202 may further receive region of interest information including a segmentation map 306 identifying pixel regions 308 of interest within the video frame 302. In particular, the segmentation map 306 may include pixel regions 308 corresponding to heads-up display (HUD) elements or other graphical elements of the video frame 302 that have been marked or otherwise identified as important regions of the display. For example, Figure 3A As shown, segment map 306 includes a plurality of pixel regions 308 corresponding to the speed indicators and player levels that appear within decompressed video frame 302. Segment map 306 may include any number of identified regions of interest.

[0048] exist Figure 3A In the example shown, the identified pixel region 308 can be identified in a variety of ways. For example, in one or more embodiments, the pixel region 308 can be identified as important by the developer of the video game content. In this case, the video content generator 106 can provide the information of the region of interest as part of the same data transmission that includes the compressed video content. Alternatively, in one or more embodiments, the region of interest manager 206 implemented as part of the client device 104 or the server device 102 can identify the information of the region of interest based on the specific content or how the content is encoded. Figure 3B-3C ) discusses in more detail additional examples regarding how to identify information about regions of interest.

[0049] like Figure 3AAs shown, the denoising system 202 can apply the denoising model to the compressed video frame 302 based on the information from the segmentation mask 306 to generate a repaired video frame 310 in which one or more compression artifacts have been removed from the decompressed video frame 302. For example, Figure 3A As shown, the denoising system 202 can specifically remove the third artifact 304c from the corresponding area of ​​pixels identified within the segmentation mask 306. In this example, the denoising system 202 removes the third artifact 304c at a location in the decompressed video frame 302 corresponding to the identified pixel area 308 within the segmentation mask 306, while not removing the first and second artifacts 304a-b from portions of the decompressed video frame 302 that do not correspond to the area of ​​interest. In this way, the denoising system 202 can selectively remove one or more artifacts from within a video frame corresponding to an area of ​​the video frame 302 identified as important in providing display of the video content.

[0050] The denoising system 202 may provide the repaired video frame 310 as an input to the super-resolution system 204. According to one or more embodiments described above, the super-resolution system 204 may upsample, interpolate, or otherwise increase the resolution of the repaired video frame 310 to generate a high-resolution video frame 312 for display via a display device. In one or more embodiments, the super-resolution system 204 includes a super-resolution model trained to receive an input image (e.g., the repaired video frame 310) and generate a higher resolution output image (e.g., the high-resolution video frame 312) for display.

[0051] Figure 3B Another example implementation is illustrated in which the interest region manager 206 is implemented as part of the video enhancement system 112 on the client device 104. For example, the interest region manager 206 can receive a decompressed video frame 320 including video content and various heads-up display (HUD) elements consistent with one or more examples discussed above. The video frame 302 can include any number of compression artifacts introduced as part of compressing and decompressing the original video content generated by the video generation system 106.

[0052] The region of interest manager 206 can analyze the content of the video frame 302 to identify one or more regions of interest of the video frame 320 and generate a segmentation mask 322 including an indication of pixels corresponding to the regions of interest. In one or more embodiments, the region of interest manager 206 includes a machine learning model trained to identify regions of interest based on different factors and parameters. For example, the region of interest manager 206 can implement a machine learning model trained to generally identify head-up display (HUD) elements across various types of video game content. This may include algorithms and models trained to identify graphical elements, such as mini-maps, floating text (e.g., text boxes that move between video frames), displayed menus, or other important content that the end user wants to be displayed as high quality as possible. In one or more embodiments, the region of interest manager 206 identifies different regions of interest for different video frames. For example, while many example regions of interest correspond to the same location of any video frame, the region of interest manager 206 can identify different regions of interest that move between frames (e.g., floating displays of text or other graphical elements).

[0053] While the region of interest manager 206 may implement a machine learning model that is trained to identify regions of interest generally (e.g., across a wide range of different types of video content), the region of interest manager 206 may further implement a machine learning model that is trained with a specific source of video content. For example, where a gaming application includes a limited number of heads-up display (HUD) interfaces, the region of interest manager 206 may implement a machine learning model that is specifically trained to identify regions of interest for a limited number of heads-up display (HUD) interfaces. In fact, in this case, rather than training the machine learning model by providing a variety of different types of video frames and identified regions of interest, the machine learning model may be targeted to video content for a specific gaming application (or other video-based application) to enable the machine learning model to accurately identify regions of interest, with significantly fewer training images than traditional machine learning models. Furthermore, training the region of interest identification model in this manner enables the client device 104 to effectively identify regions of interest for incoming video frames without having to analyze the entire video frame to predict likely locations of regions of interest across the entire image.

[0054] In one or more embodiments, the region of interest manager 206 implements a machine learning model that is received from the server device 102 and periodically updated based on further training. For example, the server device 102 may provide the machine learning model to the client device 104 for implementation on the client device 104 (e.g., Figure 3B), and periodically updates the machine learning model based on the generated video frames and the information about the relevant regions of interest received over time. The server device 102 may further provide an updated machine learning model that facilitates the region of interest manager 206 to more accurately identify the regions of interest over time. Along similar lines, the server device may further update and provide updates to other machine learning models implemented on the video enhancement system 112 (e.g., denoising model, super-resolution model).

[0055] Similar to one or more embodiments described above, the decompressed video frame 320 may be provided to the denoising system 202 for repairing one or more compression artifacts. The denoising system 202 may further receive the segmentation map 322 and selectively remove compression artifacts from portions of the decompressed video frame 320 corresponding to regions of interest identified within the segmentation map 322. Based on the decompressed video frame 320 and the segmentation map 322, the denoising system 202 may generate a repaired video frame 324 from which one or more compression artifacts have been removed. The super-resolution system 204 may further generate a high-resolution video frame 326 for display via a graphical user interface of a display device.

[0056] Figure 3C Another example implementation is illustrated in which the region of interest manager 206 is implemented as part of the encoder system 108 (eg, on the server device 102). Figure 3C As shown, raw video content 332 (e.g., generated by video generation system 106) can be provided as input to multiple components of encoder system 108. Specifically, encoder 330 can receive raw video content 332 and compress raw video content 332 to generate encoded video content 334 to provide to client device 104. For example, similar to the above combined Figure 2 In the example discussed, the encoder 330 may generate encoded video content 334 and provide it as input to the decoder 110, which decompresses the encoded video content 334 to generate a decompressed plurality of video frames 338 having one or more artifacts thereon. The decoder 110 may further provide the decompressed video frames 338 as input to the video enhancement system 112 for removing the artifacts and generating high quality video frames 340.

[0057] In addition to the above-described process of generating encoded video content, the encoder system 108 may further include a region of interest manager 206 implemented thereon that identifies information of regions of interest from the raw video content 332. For example, in one or more embodiments, the region of interest manager 206 analyzes the video content 332 to identify information of heads-up display (HUD) elements, locations of display tests, user input elements, and other regions of interest that are provided to the video enhancement system 112 for selectively removing compression artifacts from the decompressed video frames 338. For example, in one or more embodiments, the region of interest manager 206 generates a segmentation mask 336 or other composition buffer based on the analyzed content from within the raw video content 332 to provide to the video enhancement system 112.

[0058] In one or more embodiments, the interest region manager 206 generates an image buffer for a range of multiple frames of the original video content 332 (e.g., where the overlay remains consistent across multiple video frames). Alternatively, the interest region manager 206 may generate different image buffers over different durations of the video content 332 as the content displayed within the video content 332 changes over time.

[0059] Furthermore, in one or more embodiments, the region of interest manager 206 generates the segmentation mask 336 using a plurality of region of interest identification techniques discussed herein. For example, the region of interest manager 206 may identify one or more regions of interest based on information provided by a developer or information received directly from the video generation system 106. Furthermore, the region of interest manager 206 may identify one or more additional regions of interest based on a variable rate shading technique that is based on a compression rate applied when generating the encoded video content 334.

[0060] In one or more embodiments, one or more of the systems described herein may apply variable rate shading analysis based on color, shading, and other content to raw video content 332. As an example, the video generation system 106 may perform a series of operations related to adding shading to an image or a series of video frames. For example, the video generation system may implement a variable rate shading system that analyzes frame data (e.g., data used to generate uncompressed video frames before encoding) to determine locations where shading can be calculated with greater or lesser precision to reduce computations without significantly reducing the quality of shading applied to the video content.

[0061] As an illustrative example, if a dark corner of a room is to be shaded, but the difference in shading between adjacent pixels is small (e.g., adjacent pixels appear to be just a similar measure of "darkness" to an end user), the variable rate shading system may decide to shade the corresponding area of ​​the video frame at a low precision rate. The region of interest manager 206 may exploit this information by assuming that the low precision shaded area is less important to the overall quality of the output video content and is therefore less important than other areas of the video frame.

[0062] Based on the variable rate shading information, the region of interest manager 206 may allocate or otherwise identify regions of interest based on the shading precision applied to different portions of the video content. For example, the region of interest manager 206 may identify regions of interest based on determining that those regions of interest are not associated with low-precision shading rates (e.g., shading rates below a predetermined threshold). In one or more embodiments, the video generation system 106 provides a variable rate shading map (e.g., a variable rate shading buffer) to the region of interest manager 206 (on a server or client device), which includes an identification of regions where low-precision shading exists. The region of interest manager 206 can utilize this information when denoising and otherwise processing decompressed video frames according to one or more implementations described herein.

[0063] While one or more of the embodiments described above relate to identifying a region of interest defined by selected pixels or regions of pixels within a digital video frame, in one or more embodiments, the region of interest may additionally or alternatively refer to one or more overlays or channels that together comprise an image frame. For example, Figure 4 4 illustrates an example where the region of interest manager 206 receives a decompressed digital video frame 402 and identifies a first color channel 404a from a plurality of color channels 404a-c as a region of interest to be provided to the denoising system 202 and / or the super-resolution system 204. Figure 4 As shown, the region of interest manager 206 identifies a first color channel 404a to provide to the denoising system 202 to remove compression artifacts from the first color channel 404a, and provides additional color channels 404b-c as inputs to the super-resolution system 204 for processing in conjunction with the repaired video frame 406 provided by the denoising system 202.

[0064] Color channels 404a-c may refer to various different parts of a video frame. As a first example, color channels 404a-c may refer to YC b C r The first color channel 404a refers to the brightness channel, the second color channel 404b refers to the first chrominance channel (e.g., the chrominance blue channel), and the third color channel 404c refers to the second chrominance channel (e.g., the chrominance red channel).b C r In the alternative of the color channel, the color channels 404a-c can refer to the discrete color channels of the RGB image. In fact, in the case where the video frame includes multiple composite images that together form the video frame, the color channels 404a-c can refer to any number of channels or parts of the composite images.

[0065] In addition, despite Figure 4 Illustrated is an example where the first color channel 404a is provided to the denoising system 202 and the second and third color channels 404b and 404c are provided to the super-resolution system, in one or more embodiments, only the first color channel 404a is provided as an input to the denoising system 202 and the super-resolution system 204, and is combined with the remaining color channels 404b-c after generating the high-resolution first channel 404a with one or more artifacts removed therefrom. In this way, the video enhancement system 112 can further preserve local processing resources of the client device 104 where possible (e.g., where the additional color channels 404b-c provide negligible benefit in providing further refinement to the denoising system 202 and / or the super-resolution 204). In fact, in one or more implementations, by removing artifacts and otherwise enhancing only a single channel (e.g., the luma channel), the video enhancement system 112 can generate a high-quality image without consuming additional processing power to enhance other channels (e.g., chroma channels) of the same video frame.

[0066] In one or more embodiments, features and functionality described in conjunction with identifying a region of interest comprising a region of pixels (e.g., in conjunction with Figure 2-3C discussed) can be combined with Figure 4 For example, in addition to identifying pixel regions from the video frames for selective enhancement, the region of interest manager 206 may additionally identify select color channels from the pixel regions to further reduce the amount of processing performed by the video enhancement system 112 when enhancing the video frames for display. In fact, by combining the various techniques discussed herein, the video enhancement system 112 can selectively enhance video frames having high bit rates while maintaining high performance and without draining the battery or data resources of the client device 104.

[0067] Now turn to Figure 5-6 , these figures illustrate an example flow chart including a series of actions for selectively enhancing video content based on an identified region of interest in the video content. Figure 5-6 Actions according to one or more embodiments are illustrated, but alternative embodiments may omit, add, reorder, and / or modify Figure 5-6 In addition, in conjunction with Figure 5 or Figure 6One or more individual features and functions described as a single act or series of acts may be similarly applied to other embodiments and examples described herein. Figure 5-6 The actions of may be performed as part of a method. Alternatively, the non-transitory computer readable medium may include a computer program that, when executed by one or more processors, causes the computing device to perform Figure 5-6 In a further embodiment, the system may execute Figure 5-6 action.

[0068] Figure 5 A series of actions 500 for selectively refining a decoded video frame according to one or more implementations is illustrated. Figure 5 As shown, a series of actions 500 includes an action 510 of generating a decompressed digital video including a plurality of decoded video frames. For example, action 510 may include decompressing compressed digital video content to generate decompressed digital video content including a plurality of decoded video frames.

[0069] like Figure 5 As further shown, the series of acts 500 includes an act of identifying a decoded video frame 520. For example, act 520 may include identifying a decoded video frame from the plurality of decoded video frames.

[0070] The series of actions 500 may further include an action 530 of identifying a region of interest of the decoded video frame. For example, action 530 may include identifying a region of interest of the decoded video frame, the region of interest comprising a portion of the decoded video frame. In one or more embodiments, identifying the region of interest comprises identifying a discrete region of one or more pixels of the decoded video frame.

[0071] In one or more embodiments, identifying the region of interest includes identifying a first color channel from a plurality of color channels that together comprise the decoded video frame. Identifying the first color channel may include identifying a luma channel from a family of color channels including a luma channel, a first chroma channel, and a second chroma channel.

[0072] In one or more embodiments, identifying the region of interest includes receiving an identification of the region of interest from a server device, together with receiving the compressed digital video from the server device. Receiving the identification of the region of interest may include receiving a composition buffer including a segmentation mask of the decoded video frame, wherein the segmentation mask includes an indication of important content within the decoded video frame. In one or more implementations, the segmentation mask indicates one or more heads-up display (HUD) elements, and the one or more heads-up display (HUD) elements include information displayed as part of a graphical user interface of the live video stream. In one or more implementations, receiving the identification of the region of interest includes receiving a variable rate shading buffer, which indicates regions in the decoded video frame associated with different shading accuracy rates.

[0073] The series of actions may also include applying a denoising model trained to remove one or more compression artifacts from the region of interest of the decoded video frame to the region of interest 540. For example, action 540 may include applying the denoising model to a portion of the decoded video frame to generate a denoised video frame in which one or more compression artifacts from the region of interest of the decoded video frame have been removed.

[0074] The denoising model may include a machine learning model trained to receive an input image including at least one compression artifact and generate an output image in which the at least one compression artifact has been removed. The denoising model may be trained based on training image pairs, the training image pairs may include a first plurality of non-compressed images and a second plurality of decompressed digital images based on the first plurality of non-compressed images after compression.

[0075] The series of actions 500 may further include applying a super-resolution model to the denoised video frame to generate an output video frame having a higher resolution than the decoded video frame. The super-resolution model may include another machine learning model that is trained to receive an input digital image having an input resolution to generate an output digital image having a higher resolution than the input resolution based on the input digital image.

[0076] The series of actions 500 may additionally include identifying a region of interest of an additional decoded video frame from the plurality of decoded video frames, wherein the region of interest includes a corresponding portion of the additional decoded video frame. The series of actions 500 may further include applying the denoising model to each of the corresponding portions of the additional decoded video frames to generate a plurality of denoised video frames in which compression artifacts from the region of interest of the decoded video frames have been removed.

[0077] Figure 6600 illustrates another example series of actions for selectively enhancing digital video content according to one or more embodiments described herein. Figure 6 As shown, a series of actions 600 may include an action 610 of receiving video content to be streamed to a client device. For example, action 610 may include receiving video content to be streamed to a client device, wherein the video content includes a plurality of non-compressed video frames. Figure 6 As further shown, the series of acts 600 may include an act 620 of compressing the video content to generate compressed digital video content.

[0078] As further shown, a series of actions 600 may include an action 630 of identifying information of an area of ​​interest for video content, the information of the area of ​​interest including an indication of one or more areas of interest. For example, action 630 may include information identifying an area of ​​interest for the video content, the information of the area of ​​interest including an indication of one or more discrete areas of content from the video content. In one or more embodiments, the information identifying the area of ​​interest includes receiving an identified important area within the received video content in conjunction with receiving the video content. The identified important area within the received video content may indicate one or more heads-up display (HUD) elements, the one or more heads-up display (HUD) elements including information displayed as part of a graphical user interface of a live video stream.

[0079] The series of actions 600 may additionally include providing the compressed digital video content to the client device an action 640. The series of actions 600 may further include providing the region of interest information along with the compressed digital video content to the client device for removing compression artifacts from the decoded video frame an action 650. Action 650 may include providing the region of interest information along with the compressed digital video content to the client device for removing compression artifacts from the identified region of pixels from the decoded video frame associated with the region of interest information.

[0080] In one or more embodiments, the information identifying the region of interest includes generating a segmentation mask including an identification of a region of the video frame corresponding to an important region within the received video content. Additionally, providing the information of the region of interest may include providing the segmentation mask to the client device for selectively removing compression artifacts from the decoded video frame.

[0081] In one or more implementations, the information identifying the region of interest includes identifying portions of the compressed digital video content associated with different shading accuracy rates. Additionally, providing the information of the region of interest may include providing information associated with the different shading accuracy rates of different portions of the compressed digital video content to the client device for selectively removing artifacts from the decoded video frame.

[0082] Figure 7 Illustrated are certain components that may be included within computer system 700. One or more computer systems 700 may be used to implement the various devices, components, and systems described herein.

[0083] Computer system 700 includes processor 701. Processor 701 may be a general purpose single-chip or multi-chip microprocessor (e.g., Advanced RISC (Reduced Instruction Set Computer) Machine (ARM)), a dedicated microprocessor (e.g., a digital signal processor (DSP)), a microcontroller, a programmable gate array, etc. Processor 701 may be referred to as a central processing unit (CPU). Figure 7 Only a single processor 701 is shown in the computer system 700 , but in an alternative configuration, a combination of processors (eg, an ARM and DSP) could be used.

[0084] The computer system 700 also includes a memory 703 in electronic communication with the processor 701. The memory 703 may be any electronic component capable of storing electronic information. For example, the memory 703 may be embodied as a random access memory (RAM), a read-only memory (ROM), a magnetic disk storage medium, an optical storage medium, a flash memory device in a RAM, an onboard memory included in a processor, an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) memory, a register, etc., including combinations thereof.

[0085] Instructions 705 and data 707 may be stored in memory 703. Instructions 705 may be executed by processor 701 to implement some or all of the functionality disclosed herein. Executing instructions 705 may involve using data 707 stored in memory 703. Any of the various examples of modules and components described herein may be implemented in part or in whole as instructions 705 stored in memory 703 and executed by processor 701. Any of the various examples of data described herein may be in data 707 stored in memory 703 and used during execution of instructions 705 by processor 701.

[0086] The computer system 700 may also include one or more communication interfaces 709 for communicating with other electronic devices. The communication interface 709 may be based on wired communication technology, wireless communication technology, or both. Some examples of the communication interface 709 include a universal serial bus (USB), an Ethernet adapter, a wireless adapter operating according to the Institute of Electrical and Electronics Engineers (IEEE) 802.11 wireless communication protocol, Wireless communication adapter and infrared (IR) communication port.

[0087] The computer system 700 may also include one or more input devices 711 and one or more output devices 713. Some examples of input devices 711 include keyboards, mice, microphones, remote control devices, buttons, joysticks, trackballs, touch pads, and light pens. Some examples of output devices 713 include speakers and printers. One specific type of output device that is typically included in the computer system 700 is a display device 715. The display device 715 used with the embodiments disclosed herein may utilize any suitable image projection technology, such as a liquid crystal display (LCD), a light emitting diode (LED), gas plasma, electroluminescence, etc. A display controller 717 may also be provided for converting data 707 stored in the memory 703 into text, graphics, and / or moving images (as appropriate) displayed on the display device 715.

[0088] The various components of computer system 700 may be coupled together by one or more buses, which may include a power bus, a control signal bus, a status signal bus, a data bus, etc. For clarity, the various buses are described in detail in the following sections. Figure 7 Illustrated as bus system 719.

[0089] Unless specifically described as being implemented in a particular manner, the techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. Any features described as modules, components, etc. may also be implemented together in an integrated logic device, or individually as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a non-transient processor-readable storage medium including instructions that, when executed by at least one processor, perform one or more methods described herein. Instructions may be organized into routines, programs, objects, components, data structures, etc., which may perform specific tasks and / or implement specific data types, and which may be combined or distributed as desired in various embodiments.

[0090] The steps and / or actions of the methods described herein may be interchanged with one another without departing from the scope of the claims. In other words, unless the proper operation of the described method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0091] The term "determining" includes a wide variety of actions, and thus, "determining" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, a database, or another data structure), ascertaining, and the like. Also, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Likewise, "determining" may also include resolving, selecting, choosing, establishing, and the like.

[0092] The terms "comprising," "including," and "having" are intended to be inclusive and mean that there may be additional elements in addition to the listed elements. Additionally, it will be understood that reference to "one embodiment" or "an embodiment" of the present disclosure is not intended to be interpreted to exclude the existence of additional embodiments that also incorporate the described features. For example, any element or feature described with respect to an embodiment herein may be combined with any element or feature of any other embodiment described herein, where compatible.

[0093] The present disclosure may be embodied in other specific forms without departing from its spirit or characteristics. The described embodiments are considered to be illustrative rather than restrictive. Thus, the scope of the present invention is indicated by the appended claims rather than the foregoing description. Changes within the meaning and scope of the equivalent schemes of the claims should be covered by the scope of the claims.

Claims

1. A method comprising: decompressing the compressed digital video content to generate decompressed digital video content comprising a plurality of decoded video frames; identifying a decoded video frame from the plurality of decoded video frames; identifying a first portion of the decoded video frame, the first portion of the decoded video frame comprising a first color channel from a plurality of color channels that collectively comprise the decoded video frame; as well as A denoising model is selectively applied to the first portion of the decoded video frame to generate a denoised video frame in which one or more compression artifacts from the first portion of the decoded video frame have been removed, wherein the denoising model comprises a machine learning model trained to receive an input image comprising at least one compression artifact and to generate an output image in which the at least one compression artifact has been removed.

2. The method according to claim 1, characterized in that The denoising model is trained based on training image pairs including a first plurality of non-compressed images and a second plurality of decompressed digital images based on the first plurality of non-compressed images after compression.

3. The method according to claim 1, characterized in that Further comprising applying a super-resolution model to the denoised video frame to generate an output video frame having a higher resolution than the decoded video frame, wherein the super-resolution model comprises a second machine learning model trained to receive an input digital image having an input resolution to generate an output digital image having a higher resolution than the input resolution based on the input digital image.

4. The method according to claim 1, characterized in that The first color channel comprises a luma channel from a family of color channels comprising a luma channel, a first chroma channel, and a second chroma channel.

5. The method according to claim 1, characterized in that The plurality of color channels include a red channel, a green channel, and a blue channel of a red-green-blue (RGB) image.

6. The method of claim 1, further comprising: identifying a first portion of an additional decoded video frame from the plurality of decoded video frames, the first portion comprising a first color channel of the additional decoded video frame; as well as The denoising model is applied to each of the respective portions of the additional decoded video frames to generate a plurality of denoised video frames in which compression artifacts from the first portion of the additional decoded video frames have been removed.

7. The method according to claim 1, characterized in that Selectively applying the denoising model includes applying the denoising model to the first color channel of the decoded video frame without applying the denoising model to one or more additional color channels that collectively comprise the decoded video frame.

8. The method of claim 1, further comprising: A segmentation mask of the decoded video frame is received, the segmentation mask comprising an identified region of interest indicating a high priority content area within the decoded video frame, wherein selectively applying the denoising model comprises applying the denoising model to the identified region of interest of the first portion of the decoded video frame.

9. The method according to claim 8, characterized in that The region of interest is identified based on one or more heads-up display (HUD) elements within the decoded video frame, the one or more heads-up display (HUD) elements including information displayed as part of a graphical user interface of a live video stream.

10. The method according to claim 8, characterized in that The segmentation mask is received from a server device in conjunction with receiving the compressed digital video from the server device.

11. A method comprising: decompressing the compressed digital video content to generate decompressed digital video content comprising a plurality of decoded video frames; identifying a decoded video frame from the plurality of decoded video frames; receiving a segmentation mask for the decoded video frame, the segmentation mask comprising a variable rate shading buffer, the variable rate shading buffer indicating one or more regions of interest within the decoded video frame based on different shading accuracy rates; as well as A denoising model is selectively applied to a first portion of the decoded video frame to generate a denoised video frame in which one or more compression artifacts from the one or more regions of interest of the decoded video frame have been removed, wherein the denoising model comprises a machine learning model trained to receive an input image comprising at least one compression artifact and generate an output image in which the at least one compression artifact has been removed.

12. The method according to claim 11, characterized in that The variable rate shading buffer is received from a server device in conjunction with receiving compressed digital video from the server device.

13. The method according to claim 11, characterized in that The variable rate shading buffer indicates the one or more regions of interest based on the one or more regions of interest having a shading accuracy rate greater than a predetermined shading accuracy rate threshold.

14. The method according to claim 11, characterized in that The denoising model is trained based on training image pairs including a first plurality of non-compressed images and a second plurality of decompressed digital images based on the first plurality of non-compressed images after compression.

15. The method according to claim 11, characterized in that Further comprising applying a super-resolution model to the denoised video frame to generate an output video frame having a higher resolution than the decoded video frame, wherein the super-resolution model comprises a second machine learning model trained to receive an input digital image having an input resolution to generate an output digital image having a higher resolution than the input resolution based on the input digital image.

16. The method according to claim 11, characterized in that Selectively applying the denoising model includes applying the denoising model to the one or more regions of interest of the decoded video frame without applying the denoising model to other portions of the decoded video frame.

17. A system comprising: one or more processors; a memory in electronic communication with the one or more processors; as well as Instructions stored in the memory, the instructions executable by the one or more processors to: decompressing the compressed digital video content to generate decompressed digital video content comprising a plurality of decoded video frames; identifying a decoded video frame from the plurality of decoded video frames; identifying a first portion of the decoded video frame, the first portion of the decoded video frame comprising a first color channel from a plurality of color channels that collectively comprise the decoded video frame; as well as A denoising model is selectively applied to the first portion of the decoded video frame to generate a denoised video frame in which one or more compression artifacts from the first portion of the decoded video frame have been removed, wherein the denoising model comprises a machine learning model trained to receive an input image comprising at least one compression artifact and to generate an output image in which the at least one compression artifact has been removed.

18. The system of claim 17, wherein: The first color channel comprises a luma channel from a family of color channels comprising a luma channel, a first chroma channel, and a second chroma channel.

19. The system of claim 17, wherein: Selectively applying the denoising model includes applying the denoising model to the first color channel of the decoded video frame without applying the denoising model to one or more additional color channels that collectively comprise the decoded video frame.

20. The system of claim 17, further comprising instructions executable by the one or more processors to: identifying a first portion of an additional decoded video frame from the plurality of decoded video frames, the first portion comprising a first color channel of the additional decoded video frame; and The denoising model is applied to each of the respective portions of the additional decoded video frames to generate a plurality of denoised video frames in which compression artifacts from the first portion of the additional decoded video frames have been removed.

Images