Enhanced multi-scale machine learning model for compressed video

Abstract

The present disclosure relates to methods, devices, systems, and non-transitory computer-readable storage media for training and using enhanced multi-scale machine learning models for compressed video.

Description

Background Art

[0001] Generally described, computing devices utilize a communication network or a series of communication networks to exchange data. Companies and organizations operate computer networks that interconnect multiple computing devices to support operations or provide services to third parties. Computing systems can be located in a single geographic location or in multiple different geographic locations (e.g., interconnected via private or public communication networks). Specifically, a data center or data processing center (generally referred to herein as a "data center") can include multiple interconnected computing systems for providing computing resources to users of the data center. A data center can be a private data center operated on behalf of an organization, or it can be a public data center operated on behalf of the public or for the public interest. Service providers or content creators (such as businesses, artists, media distribution services, etc.) can use one or more data centers to distribute content (such as websites, web content, or other digital data) to users or clients. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Various examples according to the present disclosure will be described with reference to the following drawings.

[0003] Figure 1 is a diagram illustrating an environment including a content distribution service / system having an encoding service / system to encode media files using a (e.g., multi-scale) machine learning model and to send the encoded media files to a viewer device according to some examples.

[0004] Figure 2 is a diagram illustrating an environment for creating, training, and using (e.g., multi-scale) machine learning models according to some examples.

[0005] Figure 3 is a diagram illustrating a video compression system including an encoder and a decoder according to some examples.

[0006] Figure 4 is a diagram illustrating a video encoder according to some examples.

[0007] Figure 5 is a diagram illustrating partitioning of a larger block (eg, a superblock) into smaller blocks (eg, code-processing blocks) according to some examples.

[0008] Figure 6 is a diagram illustrating a video decoder according to some examples.

[0009] Figure 7 is a high-level architecture diagram for multi-scale (e.g., full-scale and half-resolution) processing using a machine learning (e.g., predictive) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples.

[0010] Figure 8 is a diagram illustrating channel stitching according to some examples.

[0011] Figure 9 is a high-level architecture diagram for progressively upsampling multi-scale (e.g., full-scale, half, and quarter resolution) processing using a machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples.

[0012] Figure 10 is a high-level architecture diagram for multi-scale (e.g., full-scale, half-, and quarter-resolution) processing using a machine learning (e.g., predictive) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples.

[0013] Figure 11 is a high-level architecture diagram for progressively downsampling multi-scale (e.g., full-scale, half, and quarter resolution) processing using a machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples.

[0014] Figure 12 is a diagram illustrating a residual block of a machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples.

[0015] Figure 13 Shown is the input-output relationship of a rectified linear unit (ReLU) operation according to some examples.

[0016] Figure 14 The input-output relationship of the sigmoid operation according to some examples is shown.

[0017] Figure 15 is a diagram illustrating a residual block of a machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples.

[0018] Figure 16 is a diagram illustrating a residual block of a machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples.

[0019] Figure 17 is a diagram illustrating fused scale blocks (using convolutional blocks) of a machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples.

[0020] Figure 18is a diagram illustrating fused scaling blocks (using residual blocks) of a machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples.

[0021] Figure 19 is a diagram illustrating a fused scaling block (using residual and convolutional blocks) of a machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples.

[0022] Figure 20 Downsampling using strided convolutions according to some examples is shown.

[0023] Figure 21 Shown is downsampling using strided convolution and input channel grouping according to some examples.

[0024] Figure 22 Shown are three different channel groupings of a convolutional layer according to some examples.

[0025] Figure 23 Upsampling by pixel shuffle is shown according to some examples.

[0026] Figure 24 Convolution kernels with diamond, horizontal, vertical, and plus-sign spatial extent shapes are shown according to some examples.

[0027] Figure 25 shows full-resolution and half-resolution processing paths according to some examples, where in the half-resolution path the group size is 2.

[0028] Figure 26 Shown is a batch normalization operation according to some examples.

[0029] Figure 27 shows the full-resolution and half-resolution processing paths according to some examples, where six half-resolution residual blocks use a group size of 6, and the remaining half-resolution residual blocks use a group size of 8.

[0030] FIG28 illustrates full-resolution and half-resolution processing paths according to some examples, with a single channel group in the half-resolution path.

[0031] Figure 29 is a diagram illustrating video code processing according to some examples that includes a machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)) and can switch between the output of the machine learning model and the output of a constrained directional enhancement filter (CDEF).

[0032] Figure 30is a diagram illustrating video coding processing according to some examples, which includes replacing deblocking and constrained directional enhancement filter (CDEF) with a machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)).

[0033] Figure 31 is a diagram illustrating video coding processing according to some examples, which includes a machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)) that takes as input the output of a constrained directional enhancement filter (CDEF).

[0034] Figure 32 is a diagram illustrating video code processing according to some examples, which includes a machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)) implemented as a post-processor.

[0035] Figure 33 Shown are syntax structures for signaling flags in a sequence header according to some examples.

[0036] Figure 34 Shown are syntax structures for signaling model parameters in a sequence header according to some examples.

[0037] Figure 35 Shown are grammatical structures for enabling (or disabling) a machine learning (e.g., predictive) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples.

[0038] Figure 36 Shown are grammatical structures for selecting a set of model parameters according to some examples.

[0039] Figure 37 Syntax structures for selecting one or more sets of model parameters according to some examples are shown.

[0040] Figure 38 Shown are syntax structures for indicating a scale parameter according to some examples.

[0041] Figure 39 A grammatical structure for indicating scale parameters for four output channels of a machine learning (e.g., predictive) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples is shown.

[0042] Figure 40 Example assignments between model parameter sets, quantization parameter (QP) values, and picture types according to some examples are shown.

[0043] Figure 41 Shown are grammatical structures for indicating sets of model parameters available for selection according to some examples.

[0044] Figure 42 Syntax structures for indicating block-level control according to some examples are shown.

[0045] Figures 43 to 44 Syntax structures for indicating block-level control according to some examples are shown.

[0046] Figure 45 A network assembly method according to some examples is shown.

[0047] Figure 46 is a flowchart illustrating the operation of a method of using a multi-scale machine learning model according to some examples.

[0048] Figure 47 is a flowchart illustrating the operation of a method of generating a modified version of a frame based on a first feature set generated by a machine learning model and an upsampled second feature set, according to some examples.

[0049] Figure 48 An example provider network environment is shown according to some examples.

[0050] Figure 49 is a block diagram of an example provider network that provides storage services and hardware virtualization services to customers according to some examples.

[0051] Figure 50 is a block diagram illustrating an example computer system that may be used in some examples.

[0052] Figure 51 Shown is a logical arrangement of a general set of components of an exemplary computing device that may be utilized according to various examples.

[0053] Figure 52 Shown are examples of environments for implementing aspects according to various examples. DETAILED DESCRIPTION

[0054] The present disclosure relates to methods, devices, systems, and non-transitory computer-readable storage media for training and using enhanced multi-scale machine learning models for compressed video. Certain examples herein incorporate neural network approaches that have the benefit of reducing compression artifacts and improving visual quality. In some examples, the network is located within a prediction loop of a video decoder or outside the prediction loop, for example as a post-processing algorithm. In some examples, the network is controlled by information received in the bitstream, and the present disclosure describes an efficient method for signaling this information. The examples herein provide the following benefits: (i) using a multi-scale approach to reduce complexity, (ii) signaling a selector in the bitstream to a decoder (or post-processor) to dynamically build a larger neural network from smaller neural networks, and / or (iii) using a combination of group and one-dimensional convolution processes to reduce the complexity of a specific example of a multi-scale machine learning model (e.g., a network).

[0055] The present disclosure relates to methods, devices, systems and non-transitory computer-readable storage media for video coding using super-resolution recovery and residual frame coding. Certain examples herein relate to video coding techniques (e.g., methods) for coding videos by incorporating upsampling and super-resolution means into a coding loop. Certain examples herein have the following two benefits: improving coding efficiency and reducing the computational complexity of the video compression system, for example, by allowing some coding operations to be performed at different spatial resolutions. In some examples, these different spatial resolutions can be changed for different frames or pictures. The examples herein provide the following benefits: (i) a method for reducing memory consumption of a decoded picture buffer, (ii) a method for performing motion vector coding and motion compensation between pictures with different spatial resolutions, and / or (iii) a method for coding residual information at a spatial resolution different from other coding processes.

[0056] In some examples, a coding mode is selected for a video encoder (e.g., having a different coding mode selectable for each macroblock of a frame), for example, a coding mode based on a video coding standard. In one example, the video coding standard is an Advanced Video Coding (AVC) standard, such as the H.264 standard. In another example, the video coding standard is an Alliance for Open Media (AOM) standard, such as AV1, AV2, or the like.

[0057] Figure 1is a diagram illustrating an environment including a content distribution service / system 100 having an encoding service / system 106 to encode a media file (e.g., an input frame 108) according to a reference picture identification code format (e.g., of one or more (e.g., composite) encoding modes) to transmit the encoded media file to a viewer device 122, according to some examples. In some examples, the video compression (e.g., of the content distribution service / system / service) includes encoding modes for certain proper subsets of the input video. The encoding modes can be according to a video transcoding (e.g., encoding) standard. The decoding modes can be according to a video transcoding (e.g., decoding) standard.

[0058] Encoding (e.g., by encoder 116) can compress a video file (e.g., input frame 108) into a plurality of compressed frames, such as one or more intra-coded picture frames (I-frames) (e.g., where each I-frame serves as a complete image), one or more predicted picture frames (P-frames or delta frames) (e.g., where each P-frame has only changes in the image from the previous frame), and / or one or more bidirectionally predicted picture frames (B-frames) (e.g., which further save space (e.g., bits) by using the differences between the current frame and previous and / or subsequent frames to specify its content). For example, where P-frames and B-frames are inter-coded pictures. In one example, each single I-frame corresponds to (e.g., is associated with) a plurality of inter-coded frames (e.g., P-frames and / or B-frames), for example, as a group of pictures (GOP). In some examples, the encoder selects one or more prediction styles for a slice (e.g., a sequence of macroblocks), such as switching I (SI) frames (e.g., slices) (e.g., containing SI macroblocks as a special type of intra-coded macroblock) to facilitate switching between coded streams, and / or switching P (SP) frames (e.g., slices) (e.g., containing P and / or I macroblocks) to facilitate switching between coded streams. In some examples, for example, a slice can be an entire frame, but an entire frame is not required to be a slice.

[0059] An encoding and / or decoding algorithm (e.g., specified by a video coding standard) can select between inter-frame and intra-frame coding for (e.g., block-like) regions of each picture (e.g., frame 108). In some examples, inter-frame coding (e.g., as indicated by "inter" mode) uses motion vectors to perform (e.g., block-based) inter-frame predictions from other pictures (e.g., frames), for example, to exploit temporal statistical dependencies between different pictures. Reference pictures (e.g., reference frames) 110 can be stored in a reference picture buffer 110A. In some examples, intra-frame coding (e.g., as indicated by "intra" mode) uses various spatial predictions to exploit spatial statistical dependencies in the source signal of a single picture (e.g., frame). In some examples, motion vectors and intra-frame prediction modes are specified for a variety of block sizes. In some examples, the prediction residual is then further compressed using a transform before it is quantized to remove spatial correlations within the transformed block, resulting in an irreversible process that generally discards less important visual information while forming a close approximation to the source samples. In some examples, the motion vector or intra prediction mode is combined with the quantized transform coefficient information and encoded using, for example, a variable length coding process or an arithmetic coding process.

[0060] The encoding and / or decoding modes (e.g., to be used for encoding and / or decoding a particular macroblock of a frame, respectively) may include one, all, or any combination of the following: direct mode, inter mode, or intra mode. Direct mode may cause encoding to be performed using inter prediction for blocks for which motion vectors are not decoded. Two examples of direct prediction modes are spatial direct prediction mode and temporal prediction mode.

[0061] In some examples, a mode has one or more sub-modes to be specified.In the same example, the same (eg, prediction) mode is used for corresponding chroma (component) and luma (component) blocks.

[0062] For example, direct modes may include skip mode (e.g., a sub-mode) and / or B-frame (e.g., B-slice) direct mode (e.g., a sub-mode). In one example, skip mode is used for P-frames (e.g., P-slices), for example, where (e.g., spatial direct prediction) motion is derived directly from previously encoded information (e.g., so no additional motion data need to be encoded for the macroblock). In one example, direct mode is used for B-frames (e.g., B-slices), for example, where (e.g., temporal prediction) motion is derived directly from previously encoded information (e.g., so no additional motion data need to be encoded for the macroblock). Previously encoded information may be stored in reference picture buffer 110A, for example, List 0 (L0) references are reference picture lists used for inter-frame prediction of P, B, or SP slices (e.g., blocks). In some examples, inter-frame prediction for P and SP slices uses (reference picture) List 0 (L0). Due to bidirectional prediction (e.g., before or after the current frame in video order), some (e.g., direct) modes may utilize two motion vectors pointing to different references. In some examples, inter prediction for B slices uses (reference picture) list 0 and (reference picture) list 1 (L1).

[0063] For example, inter-mode (e.g., sub-mode) may include (e.g., luma) block partition sizes such as 16x16, 16x8, 8x16, or 8x8 (pixel x pixel). Inter-mode may use transforms such as 4x4 transforms or 8x8 transforms.

[0064] For example, intra modes (e.g., sub-modes) may include (e.g., luma) block partition sizes such as intra4×4, intra8×8, and intra16×16. For example, intra4×4 may further include prediction sub-modes of vertical, horizontal, DC, diagonal left below, diagonal right below, vertical right, horizontal down, vertical left, and / or horizontal up.

[0065] Coding modes may be used to encode a particular slice of a frame, e.g., where a slice is a spatially distinct region of a frame that is encoded separately from any other region in the same frame, and / or where a slice is a plurality of macroblocks (e.g., a sequence of macroblock pairs).

[0066] The encoding mode (e.g., of encoder 116) can be separate from encoder settings, for example, separate from the value settings of one, all, or any combination of the following in the encoder: spatial adaptive quantization strength, temporal adaptive quantization strength, flicker reduction, motion group of pictures (GOP) on / off, number of B frames (e.g., per GOP), direct mode (e.g., allowing B frames to use predicted motion vectors instead of actual coding of motion for each frame) (e.g., for scenes), pre-filter on / off, differential quantization parameter (QP) offset (e.g., between I frames and P frames / B frames), rate-distortion optimized quantization (RDOQ), speed settings, or additional configuration (e.g., encoder) settings.

[0067] In some examples (e.g., at the beginning of the video encoding process), the content distribution service / system / service will, for example, select a coding mode for each macroblock (or slice) of a frame. This may include a mode selection that selects a single mode (e.g., optimal from a visual quality perspective) by cycling through all available modes in the following manner: encoding according to the mode (e.g., by encoder 116), then decoding (e.g., by decoder 118), and measuring the quality between the encoded media (e.g., macroblock) and the decoded version.

[0068] In some examples (e.g., for composite mode), the encoder 116 encodes the frame 108 and sends it to the decoder 118 to decode the encoded frame. In some examples, the decoder 118 reconstructs a version of the frame 108 from the bitstream. In some examples, one or more of the decoded frames generated by the decoder 118 from the encoder 116 are input into a reference (e.g., decoded) picture buffer 110A (e.g., a decoded frame buffer / list or a reference frame buffer / list). In some examples, the reference frames 110 in the picture buffer 110A (e.g., which may be fewer than all frames in the video) are used to encode the input frame 108, for example, using previously decoded reference frames 110 via inter-frame prediction (e.g., prediction values) of the current frame.

[0069] Some (e.g., AOM) coding standards (e.g., codecs) allow a maximum number (e.g., eight frames) in their reference picture buffer 110A. In some examples, to encode frame 108, encoder 116 may select a proper subset (e.g., seven) of frames from reference picture buffer 110A as its reference frames. In some examples, the bitstream allows encoding service / system 106 to explicitly assign a unique reference frame index (e.g., ranging from 1 to 7) to each reference. In some examples, reference frame indices 1–4 are assigned to frames preceding the current frame in display (e.g., picture or video) order, while indices 5–7 are used for reference frames immediately following the current frame. In some examples of compound inter-frame prediction, two references may be combined to form a prediction. In some examples, if both reference frames precede or follow the current frame, this is a unidirectional compound prediction, in contrast to, for example, bidirectional compound prediction, in which there is one previous reference frame and one future reference frame in display (e.g., picture or video) order. In some examples, the encoding service / system 106 (e.g., its code processing standard) links the reference frame index to any frame in the decoded frame buffer, e.g., this allows it to fill all reference frame indexes when there are not enough reference frames on either side. In some examples, e.g., when frame coding is completed, the encoding service / system 106 decides which (if any) reference frame in the reference picture buffer 110A to replace and explicitly signals this in the bitstream. In some examples, the encoding service / system 106 allows bypassing updating the reference picture buffer 110A, e.g., for high motion video where some frames are not well correlated with neighboring frames.

[0070] In some examples, reference picture buffer 110A updates are implemented at the frame level through two syntaxes: (1) a multi-bit (e.g., eight-bit) reference refresh flag, e.g., where each bit signals whether the corresponding frame in the reference picture buffer 110A is to be refreshed with a newly coded frame, and / or (2) a virtual index map, where each of the reference frames is tagged with a unique virtual index, and both the encoder 116 and the decoder 118 maintain a reference frame map that associates the virtual index with a corresponding physical index pointing to its location within the reference picture buffer 110A. In some examples, both the refresh flag and the virtual index are written to the bitstream, e.g., such a mapping mechanism is used to avoid memory copies when the reference frames are being updated.

[0071] In some examples, the encoding service / system 106 includes a field 114 that, when set, causes the encoding service / system 106 (e.g., encoder 116 and / or decoder 118) to utilize the functionality discussed herein, for example, to enter a particular (e.g., multi-scale) machine learning mode. In a certain example, the decoder 118 includes, for example, one or more machine learning (e.g., prediction) models 112 (e.g., a multi-scale convolutional neural network (MSCNN)) for generating predictions according to the present disclosure.

[0072] The depicted content distribution service / system 102 includes a content data repository 104, which can be implemented in one or more data centers. In one example, an encoder 116 (e.g., its media file (e.g., segment) generator) accesses media files (e.g., video files to be viewed by a viewer device 122) (e.g., from the content data repository 104 or directly from a content provider 136, e.g., as a live stream). In some examples, the content distribution service / system 102 includes a video ingestion service 138 for ingesting video, e.g., from the content provider 136.

[0073] In some examples, a (e.g., client) viewer device 122 that requests a media file (e.g., a media segment) from the content distribution service / system 102 causes the encoder 116 to encode the video file, e.g., into a compressed format, for transmission to the viewer device 122 over the network 120. In one example, a media file generator of the encoder 116 generates one or more subsets (e.g., frames, segments, sections, scenes, etc.) of a media file (e.g., a video), e.g., starting from accessing the media file and generating the requested media (e.g., segments). In one example, each segment includes multiple video frames.

[0074] exist Figure 1 In , content distribution service / system 102 is coupled to viewer devices 122 and user devices 130 via one or more networks 120, such as a cellular data network or a wired or wireless local area network (WLAN).

[0075] In some examples, the content distribution service / system 102 (e.g., its encoding service / system 106) will send a query to, for example, a user (e.g., operator) device 130 for a desired model (e.g., a plurality of different corresponding machine learning models (e.g., as in Figures 3 to 47The user device 130 (e.g., in response to a command from a user of the device 130) will send a response (e.g., an indication of the mode 114). The depicted user device 130 includes a display 132 having a graphical user interface (GUI) 134, for example, to display information regarding the encoding service / system 106 entering (or not entering) a particular mode 114 (e.g., a plurality of different corresponding machine learning modes (e.g., as in Figures 3 to 47 One or more of the queries in ).

[0076] The depicted viewer device 122 (e.g., where the viewer is a customer or user (e.g., operator) of the device 130) includes a media player 124 having a decoder 126 (e.g., separate from the decoder 118 of the encoding service / system 106) for decoding a media file (e.g., a clip) from the content distribution service / system 102 to, for example, display the video and / or audio of the media file on a display and / or audio output 128, respectively. In a certain example, the decoder 126 includes, for example, one or more machine learning (e.g., prediction) models 112 (e.g., a multi-scale convolutional neural network (MSCNN)) for generating predictions in accordance with the present disclosure. In some examples, the decoder 126 (e.g., as code and / or hardware) includes a reference (e.g., decoded) picture buffer 126A. In some examples, decoder 126 receives an indication (e.g., a syntax element in a bitstream) of the type of identification code and / or number of reference slots (e.g., reference frames in a reference picture list) that can be used for the composite mode from a media file (e.g., within a header of the media file, such as a sequence and / or picture header of the coded media). In some examples, any encoder and / or decoder (e.g., decoder 126) should be aware of the format of the "reference picture identification code" used. In some examples, decoder 126 decodes an encoded frame (e.g., a picture) based on (i) the already decoded (e.g., reference) frames in its reference (e.g., decoded) picture buffer 126A and (ii) the identification code of the reference frame for use in decoding the current frame (e.g., and the format of the "reference picture identification code"). In some examples, the decoded current frame is then played by media player 124, e.g., displayed on display 128.

[0077] In some examples, the viewer device 122 includes a post-processor, for example, for performing post-processing operations. In some examples, the post-processing operations include executing one or more machine learning (e.g., prediction) models 112 (e.g., multi-scale convolutional neural network (MSCNN)), for example, for generating predictions according to the present disclosure. In some examples, the post-processor 140 is separate from the decoder (or encoder), for example, so that support for one or more machine learning (e.g., prediction) models 112 (e.g., multi-scale convolutional neural network (MSCNN)) can be added to an encoder (e.g., standard) or decoder (e.g., standard) (e.g., codec) that does not include and / or support machine learning.

[0078] Figure 2 is a diagram illustrating an environment for creating, training, and using one or more machine learning models 112 according to some examples. Figure 2 Included are a video compression service 146, one or more storage services 216, one or more machine learning services 230, and one or more computing services 242 implemented within a multi-tenant provider network 200. Each of the video compression service 146, one or more storage services 216, one or more machine learning services 230, one or more model training services 232, one or more hosting services 234, and one or more computing services 242 can be implemented via software, hardware, or a combination of both, and can be implemented in a distributed manner using multiple different computing devices.

[0079] The provider network 200 (or "cloud" provider network) provides users with the ability to utilize one or more of various types of computing-related resources, such as computing resources (e.g., executing virtual machine (VM) instances and / or containers, executing batch jobs, executing code without provisioning servers), data / storage resources (e.g., object storage, block-level storage, data archive storage, databases and database tables, etc.), network-related resources (e.g., configuring virtual networks (including multiple groups of computing resources), content delivery networks (CDNs), domain name services (DNS)), application resources (e.g., databases, application build / deployment services), access policies or roles, identity policies or roles, machine images, routers, and other data processing resources. These and other computing resources may be provided as services, such as hardware virtualization services that can execute computing instances or serverless code execution services that can execute code (either of which may be referred to herein as computing services 242), storage services 216 that can store data objects, etc. Users (or "customers") of the provider network 200 may utilize one or more user accounts associated with a customer account, but these terms may be used somewhat interchangeably depending on the context of use. Users may interact with the provider network 200 over one or more intermediate networks 206 (e.g., the Internet) via one or more interfaces, such as by using application programming interface (API) calls, via a console 205 implemented as a website or application, etc. The interfaces may be part of, or serve as a front end to, the control plane of the provider network 200, which includes "back-end" services that support and enable services that might be provided more directly to customers.

[0080] For example, a cloud provider network (or just "the cloud") generally refers to a large pool of accessible virtualized computing resources (such as computing, storage and networking resources, applications, and services). A cloud can provide convenient, on-demand network access to a shared pool of configurable computing resources that can be programmatically provisioned and released in response to customer commands. These resources can be dynamically provisioned and reconfigured to adjust to variable loads. Thus, cloud computing can be considered to be both applications delivered as services over publicly accessible networks (e.g., the Internet, cellular communication networks) and the hardware and software in the cloud provider's data center that provides these services.

[0081] Typically, the traffic and operations of a provider network can be broadly broken down into two categories: control plane operations carried on the logical control plane and data plane operations carried on the logical data plane. The data plane represents the movement of user data through a distributed computing system, while the control plane represents the movement of control signals through a distributed computing system. The control plane typically includes one or more control plane components distributed across and implemented by one or more control servers. Control plane traffic typically includes management operations such as system configuration and management (e.g., resource placement, hardware capacity management, diagnostic monitoring, system status information). The data plane includes customer resources implemented on the provider network (e.g., compute instances, containers, block storage volumes, databases, file storage). Data plane traffic typically includes non-management operations, such as the transfer of customer data to and from customer resources. Control plane components are typically implemented on a set of servers separate from the data plane servers, and control plane traffic and data plane traffic can be sent over separate / different networks.

[0082] To provide these and other computing resource services, the provider network 200 typically relies on virtualization technology. For example, virtualization technology can be used to provide users with the ability to control or utilize computing instances (e.g., VMs using a guest operating system (O / S) that operates using a hypervisor that may or may not further operate on top of an underlying host O / S; containers that may or may not operate within a VM; instances that can execute on "bare metal" hardware without an underlying hypervisor), where a single electronic device can be used to implement one or more computing instances. Thus, users can directly utilize computing instances hosted by the provider network (e.g., provided by a hardware virtualization service) to perform various computing tasks. Additionally or alternatively, users can indirectly utilize computing instances by submitting code to be executed by the provider network (e.g., via an on-demand code execution service), which in turn utilizes the computing instances to execute the code, typically without the user having any control or knowledge of the underlying computing instances involved.

[0083] For example, in various examples, a "serverless" function may include code that is provided by a user or other entity (such as the provider network itself) and can be executed on demand. The serverless function may be maintained within the provider network 200 via an on-demand code execution service (which may be one of the computing services 242) and may be associated with a specific user or account or generally accessible to multiple users / accounts. The serverless function may be associated with a uniform resource locator (URL), uniform resource identifier (URI), or other reference that can be used to invoke the serverless function. The serverless function may be executed by a computing instance (such as a virtual machine, container, etc.) when triggered or invoked. In some examples, the serverless function may be invoked via an application programming interface (API) call or a specially formatted hypertext transfer protocol (HTTP) request message. Thus, a user can define a serverless function that can be executed on demand (e.g., as an application 240B) without requiring the user to maintain dedicated infrastructure to execute the serverless function. Alternatively, the serverless function may be executed on demand using resources maintained by the provider network 200. In some examples, these resources may be maintained in a “ready” state (e.g., with a pre-initialized runtime environment configured to execute the serverless function), thereby allowing the serverless function to be executed in near real time.

[0084] In some examples, the video compression service 146 is a machine learning driven service that generates one or more predictions for video compression, e.g., as described in reference Figures 3 to 47 What is being discussed.

[0085] For example, training system 250 may enable a user to generate one or more machine learning models (e.g., multi-scale machine learning model 112).

[0086] Examples herein allow for creation of one or more machine learning models 112 by supplying a training dataset 218 (e.g., including labels 220).

[0087] In some examples, video compression service 146 allows users to create and use models 112 via use of custom model system 208 .

[0088] At a high level, machine learning can include two main components that need to be in place in order to display advertised functionality to customers: (i) training and (ii) inference. Training can include the following responsibilities: training data analysis; data splitting (training, evaluation (e.g., development or validation), and / or test data); model selection; model training; model evaluation; and status reporting. Inference can include the following responsibilities: model loading and hosting; and inference (e.g., synchronization and batching).

[0089] Training can include training the candidate algorithm into a model (e.g., machine learning model 112) and corresponding configurations (e.g., coefficients and / or hyperparameters). Training can perform a grid search on a (e.g., pre-defined) experiment matrix to search for the model and its parameters (e.g., hyperparameters) that perform best on a given dataset.

[0090] Thus, user 209 may provide or otherwise identify data 218 for use in creating a custom model (e.g., using tags 220). For example, as shown at circle (1), user 209 may utilize client application 203 executed by computing device 204 (e.g., a web application that implements console 205 for provider network 200, a standalone application, another web application of another entity that utilizes classification service 146 as part of its backend, a database or hybrid SQL environment, etc.) to cause computing device 204 to upload data 218 to a storage location (e.g., provided by storage service 216 such as an object storage service of provider network 200).

[0091] Data 218 may be a columnar data set including rows (or entries) of data values, where the data values may be arranged according to one or more columns (or attributes) and may be of the same data type (e.g., a data type storing text). In some cases, data 218 includes headers or other metadata describing the names or data types of the columns, but in some cases, this metadata may not be present. For example, some or all of data 218 may have been provided by a user as a plain text file (e.g., a comma-separated value (CSV) or tab-separated value (TSV) file), an exported database table or structure, an application-specific file (such as a spreadsheet), etc.

[0092] For example, when the user 209 desires to train the model 112, the file (or files) may include tags corresponding to the files (e.g., video, audio, and / or text), such as tags with categories indicating the content in the files.

[0093] Thereafter, at circle (2), computing device 204 may issue one or more requests (e.g., API calls) to machine learning service 230 indicating that user 209 desires to train one or more algorithms into a model (e.g., machine learning model 112). The request may be an identifier of which type of model to create or an identifier of the type of candidate model (e.g., candidate machine learning model 112) that machine learning service 230 itself will identify. The request may also include one or more of the following: an identifier of one or more storage locations where data 218 is stored (e.g., an identifier of label 220), which may identify a storage location within provider network 200 (e.g., as provided by storage service 216) or external to provider network 200 (e.g., via a uniform resource locator (URL), bucket / folder identifier, etc.); a format identifier for data 218; a language identifier for the language of label 220, etc. In some examples, the request includes an identifier of the candidate algorithm itself within the request (e.g., from user 209). In some examples, storage service 216 stores input files 222, e.g., videos 224 and / or images 226.

[0094] In response to receiving the request, the custom model system 208 of the machine learning service 230 is called and begins operations for training a model of the corresponding type. For example, the custom model system 208 can identify what type of model to train (e.g., by analyzing a method call associated with the request), a storage location associated with the data 218 (e.g., labels 220), etc. Thus, the custom model system 208 can retrieve any stored data 218 element, as shown at circle (3), which can be from a storage location within the provider network 200 or external to the provider network 200.

[0095] In some examples, training of model 112 (at dashed circle (4) in model 112) includes performing (at optional dashed circle (4)) specific training jobs (e.g., hyperparameter optimization tuning jobs) by training service 232 of machine learning service 230, among other things.

[0096] In some examples, the hosting system 252 of the custom model system 208 (at circle (5)) can utilize (at optional dashed circle (5)) the hosting service 234 of the machine learning service 230 to deploy the model as a hosted model 236 in association with an endpoint 238, which can receive inference requests from client applications 240A and / or 240B at circle (8), provide inference requests 260A to the associated hosted model 236, and provide inference results 260B (e.g., predictions) back to the application 240A and / or 240B, which can be executed by one or more computing devices 207 external to the provider network 200 or by one or more computing devices of a computing service 242 (e.g., a hardware virtualization service, a serverless code execution service, etc.) within the provider network 200. The inference results 260B can be displayed to a user and / or viewer (e.g., in a graphical user interface of the application) and / or exported as a data structure (e.g., in a selected format). In some examples, the inference results are utilized by the encoding service / system 106 .

[0097] Overview

[0098] Examples herein relate to methods for enhancing compressed video. In some examples, the methods incorporate a neural network approach that has the benefit of reducing compression artifacts and improving visual quality. The network can be located within the prediction loop of a video decoder or outside the prediction loop as a post-processing algorithm. In some examples, the network is controlled by information received in the bitstream, and efficient methods for signaling this information are disclosed herein. Other key benefits of the approach include: (i) using a multi-scale approach to reduce complexity, (ii) signaling a selector in the bitstream to the decoder or post-processor to dynamically build a larger neural network from smaller neural networks, and (iii) specific examples of networks that use a combination of group and one-dimensional convolutional processes to reduce complexity.

[0099] Video Compression

[0100] In some examples, the video compression system includes video encoding, video decoding, and video post-processing operations. In some examples, the video encoder receives one or more images (or equivalently, frames or pictures) having one or more color channels as input and generates a bitstream as output. In some examples, the video decoder receives all or part of the bitstream as input and generates one or more images as output. These output pictures are similar to the images received by the encoder, but may not be identical. The video post-processor is optional and receives the pictures generated by the decoder as input and generates enhanced pictures as output. Figure 3 An example video compression system (eg, an overview of video compression system 300 ) is shown in .

[0101] Figure 3 is a diagram illustrating a video compression system 300 including an encoder 304 and a decoder 308, according to some examples. In some examples, encoder 304 is an instance of encoder 116. In some examples, decoder 308 is an instance of decoder 126. In some examples, decoder 308 is an instance of decoder 118.

[0102] In some examples, encoder 304 receives input having an image (e.g., a frame of video) and generates output having a bitstream 306 (e.g., a coded bitstream of video). In some examples, decoder 308 receives input having a bitstream 306 (e.g., a coded bitstream of video) and generates output having a decoded image 310 (e.g., a decoded frame of video). In some examples, video compression system 300 outputs an enhanced image 314. In some examples, (optional) post-processor 312 receives input having a decoded image 310 (e.g., a decoded frame of video) and generates output having an enhanced image 314 (e.g., an enhanced decoded frame of video).

[0103] The video compression system may use a video coding standard (e.g., H.264, HEVC, VVC, VP9, or AV1 standard) to describe one or more of a bitstream, a decoder, an encoder, or a post-processor. In some examples, the video coding standard defines the construction and / or decoding process of the bitstream. Figure 4 An example video encoder is shown in .

[0104] Figure 4 is a diagram illustrating a video encoder 304 according to some examples. Figure 4 As can be seen in FIG, the encoder 304 receives an image as input and a splitting operation 402 divides the image into spatial regions for code processing. These spatial regions may be referred to as macroblocks, superblocks, code processing tree units, or other terms known to those skilled in the art. In some examples, the spatial regions are then further partitioned. For example, each superblock (e.g., in AV1) may be recursively split into code processing blocks of varying sizes (e.g., from 128×128 samples to 4×4 samples) and / or having a square and / or rectangular shape. Furthermore, spatial regions may be combined into larger spatial regions, referred to as tiles, slices, or other terms known to those skilled in the art.

[0105] For color channels, both can be done jointly or independently. Figure 5 An example of partitioning a shape (eg, partitioning a superblock into code processing blocks) is shown in . Figure 5is a diagram illustrating partitioning a larger block (e.g., superblock) 502 into smaller blocks (e.g., code processing blocks) according to some examples. In some examples, a sample (or pixel) corresponds to a specific location and color channel within a frame. For a two-dimensional image, the specific location can be a horizontal and vertical index into a color channel of the frame, for example, which stores the value of the image at that index.

[0106] Return to Figure 4 In some examples, each code-processed block is first predicted at 404 using intra prediction, inter prediction, or a combination thereof. In some examples, intra prediction 406 predicts the current code-processed block from previously code-processed and spatially adjacent blocks. This prediction can be performed using directional intra prediction, which predicts sample values of the current code-processed block by extrapolating information from previous code processing along a prediction direction. This prediction can also be performed using non-directional intra prediction, such as non-directional smoothing intra prediction, recursive intra prediction, intra block copying, and palette techniques.

[0107] In some examples, inter-frame prediction 408 uses information from previously coded frames stored in one or more frame buffers for prediction. One method for performing this prediction uses a translational motion model. With this approach, a region of the previously coded frame is translated using the spatial offset (or motion vector) between the current coded block and the previously decoded frame, and the translated version is used for prediction. Different accuracies of the motion vectors are possible, such as 1 / 8 pixel motion vector accuracy. In addition, different interpolation filters can also be selected. In addition to the translational motion approach, alternative methods (or prediction models) for performing inter-frame prediction include affine motion compensation and overlapped block motion compensation. In addition, one or more of these models can predict the current coded block from more than one previously coded position in the previously decoded frame. One example is the composite prediction mode in AV1. Strategies for combining more than one prediction include calculating a weighted average based on the temporal distance between each previously coded block and the current coded block. In the case where the previously coded frame is of a different resolution than the input frame, the sampler can optionally convert the spatial resolution of the previously coded frame.

[0108] In some video code processing systems, a combination of intra-frame prediction and inter-frame prediction may be used for the block currently being processed. For example, the code processing block may be divided into two regions. The first region is predicted using an intra-frame prediction method, while the second region is predicted using an inter-frame prediction method. As a second example, the intra-frame prediction and inter-frame prediction may be averaged (e.g., via a weighted average) to predict the current code processing block.

[0109] After each block is predicted, residual information can be added to the prediction at 410. The encoder 304 can first calculate the difference between the prediction and the original frame data 302, apply an optional transform 412 to the difference, and quantize 414 the coefficients output by the transform. In some examples, at both the encoder and the decoder, the residual is calculated by dequantizing 416 (e.g., inverse quantizing) the quantized coefficients calculated by the encoder, applying an optional inverse transform 418 to the dequantized coefficients, and adding the results of the inverse transform to the predicted block at 420. Note that the sequential process of quantization and dequantization may not produce the same output as the input provided to the quantization process. Similarly, the sequential process of transform followed by inverse transform may not produce the same output as the input provided to the transform.

[0110] The reconstructed block corresponding to the addition of prediction and residual information may then be processed by one or more in-loop filters 422 (or operations). In some examples, these filters improve the fidelity of the reconstructed block and may include processes such as a deblocking filter 424, a constrained directional enhancement filter (CDEF) 426, a sample adaptive offset filter 428, an adaptive loop filter 430, and / or a loop restoration filter 432. These operations may use different partitions than the reconstructed block.

[0111] In some examples, the output of one or more loop (e.g., in-loop) filters (e.g., improved images) 422 is stored in a frame buffer 434 (or decoded picture buffer) for use in inter-frame prediction of code-processed blocks in different frames. In some examples, the frame buffer 434 is Figure 1 10A in the decoded picture buffer. In addition, the output can be processed by out-of-loop filters (or operations) to further modify the output. Examples of these filters (or post-processing filters) include spatial resizing, color conversion, film grain synthesis, and de-banding operations. In some examples, the result is not stored in the decoded picture buffer.

[0112] Information calculated during the encoding process can be signaled in the bitstream 306. For example, partitions of regions for code processing, intra-frame prediction directions, motion vectors, quantized transform coefficients, and in-loop filter control information can be signaled. In some examples, this information is sent (e.g., losslessly) using an entropy code processing system (e.g., an entropy encoder 436). In some examples, the encoder 436 takes as input information from one or more of the depicted operations, such as the quantized values output from the quantizer 414. In some examples (e.g., AV1), the entropy code processing system uses an M-ary arithmetic code processor. In some examples (e.g., VVC), the entropy code processing system uses a context-adaptive binary arithmetic code processor. In some examples, the decoder then extracts the information from the bitstream.

[0113] Figure 6 is a diagram illustrating a video decoder 308 according to some examples. As described above, in some examples, the video decoder 308 takes the coded bitstream 306 as input and decodes the bitstream using an entropy decoder 602. In some examples, the entropy decoder 602 generates quantized coefficients as output and also controls information used for other operations within the decoder. In some examples, the quantized coefficients are inversely quantized at 604 and (optionally) inversely transformed at 606 to generate a residual. In some examples, the residual is added to a block-level prediction generated by an intra-frame prediction, inter-frame prediction, or combined prediction process 610 at 608. In some examples, after addition, the resulting sample values are processed by a loop filter 612. Example loop filter 612 operations include one or any combination of deblocking 614, a constrained directional enhancement filter (CDEF) 616, a sample adaptive offset 618, an adaptive loop filter 620, and / or a restoration filter 622. In some examples, the loop filter output is stored in one or more frame buffers 624, for example, to be used by an inter-frame prediction process and / or provided as an output from the decoder 308. In some examples where the data stored in the frame buffers 624 does not have the same spatial resolution as the current frame, the data stored in the frame buffers can be resampled to the same resolution as the current frame by the inter-frame prediction process. In some examples, the frame buffers 624 are Figure 1 In some examples, a decoder implementation takes the coded bitstream 306 as input and then uses the bitstream (or information based on the bitstream) to generate a residual and a reconstructed frame, e.g., to generate input 701 (e.g., x' and the residual).

[0114] Problem Statement

[0115] Some video code processing systems use loop filters to improve code processing efficiency. These filters increase the quality of each decoded picture, and because the filter is in the loop, the improvement is propagated to subsequent frames using a motion compensation process. Although some standards (e.g., AV1 and VVC) can use complex means, the use of residual neural networks in decoders and / or post-processors can provide further improvements in code processing efficiency. Unfortunately, the complexity of these networks is less than desirable. In addition, some networks are fixed and cannot be reconfigured in the bitstream.

[0116] Video enhancement methods

[0117] Figure 7 is a high-level architecture diagram 700 for multi-scale (e.g., full-scale and half-resolution) processing using a machine learning (e.g., predictive) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples. Figure 7 (High-level architecture diagram of multi-scale processing) is an example of some of the methods disclosed herein. In some examples, the method operates at multiple spatial resolutions. This has the benefit of reducing the number of multiply-accumulate operations (MACs) per pixel because the number of samples to be processed at the lower resolution is smaller. For example, processing at half the spatial resolution reduces the number of samples by a quarter compared to processing at full resolution. Figure 7As can be seen in FIG, some examples herein take as input 701 an image (e.g., a frame or a proper subset of a frame) represented by x' and a residual signal. In some examples, the input 701 (e.g., a frame or a proper subset of a frame) is downsampled, e.g., the same input (x', residual) is input to both the high-resolution path and the lower-resolution downsampler. In some examples, some portions (e.g., blocks) of the input 701 are spatially downsampled 702 in the lower (e.g., half) resolution path and processed using a series of residual blocks 704A, and some portions (e.g., blocks) of the input 701 (that are not downsampled) are processed in the full-resolution path using a series of residual blocks 704B. In some examples, the outputs of these residual blocks 704A are upsampled at 706 and concatenated at 708 with, for example, the output of a series of high-resolution residual blocks 704B and a convolution (e.g., conv2d k3 nS0) operation 712. In some examples, the stitched result is provided to a fused scale operation 710 that converts the low-resolution and high-resolution data into a prediction value 714 for each sample position. In some examples, different channels of the same block (e.g., luma and chroma) are processed on different paths, for example, where a first channel (e.g., luma) of the block is processed on a full-resolution path (e.g., by a series of residual blocks 704B), and a second channel (e.g., chroma) of the same block is processed on a lower (e.g., ½) resolution path (e.g., by a series of residual blocks 704A). In some examples, this allows for power and processing savings on the lower-resolution path, for example, compared to also performing processing of the second channel (e.g., chroma) of the same block at full resolution.

[0118] In some examples, the predicted value is a feature value (e.g., a feature for each of the red, green, and blue channels of the image). In some examples, the predicted value is a change in pixel value (e.g., a difference), e.g., to make a desired correction. In some examples, the features are machine learning features determined, e.g., for a particular machine learning architecture. In some examples, each channel is a luma (e.g., brightness) value. In some examples, each channel is a chroma (e.g., color) value. In some examples, a feature set (e.g., a feature map) is generated for each channel. In some examples, there are channels for edges, textures, blocking artifacts of motion, out-of-order features, etc. In some examples, the depth of the convolution matrix in a convolution operation (e.g., a network) is the total number of channels (e.g., the same number of channels as the input).

[0119] In some examples, the convolution operation 712 applies a two-dimensional (2D) convolution to an input value consisting of several input planes. In some examples, the convolution operation 712 applies a two-dimensional (2D) convolution to an input value consisting of several input planes. in ,H,W) and output (N,Cout ,H out ,W out )’s output value is:

[0120]

[0121] where ★ is a valid 2D cross-correlation operator, N is the batch size, C represents the number of channels, H is the height of the input plane in terms of pixels or samples, W is the width in terms of pixels or samples, weight (Tensor) is the learnable weight (e.g., of a shape module), and bias (Tensor) is the learnable bias (e.g., of a shape module).

[0122] In some examples, prediction 714 is an improved set of pixels (or codec parameters), eg, a correction (or delta) to a pixel.

[0123] In some examples, Figure 7 The machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)) depicted in (or other figures) is included as Figure 1 The ML model 112 in (eg, the encoding service / system 106 and / or the decoder 126 (eg, of the viewer device 122)).

[0124] Figure 8 is a diagram illustrating channel stitching according to some examples.

[0125] like Figure 8 As can be seen in , multiple features (or tensors) with the same spatial resolution (e.g., width and height in pixels) (shown as input 0 with four channels, input 1 with three channels, and input 2 with two channels) are simply concatenated at 708 to output features with as many channels as the sum of the channels in the input features (shown as one output with 9 channels (4+3+2)). Figure 8 In some examples, each data plane is a feature, and / or the different planes are channels.

[0126] In another example, the method may use more than two scales (eg, resolutions). Figure 9 900 is a high-level architecture diagram for progressively upsampling multi-scale (e.g., full-scale, half, and quarter resolution) processing using a machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples.

[0127] For example, Figure 9 (High-Level Architecture Diagram of Multi-Scale Processing) shows an example of using three scales (e.g., resolutions) corresponding to full-resolution processing, half-resolution processing, and quarter-resolution processing.

[0128] The downsampling may use other downsampling factors, and may differ in the horizontal and vertical (or, alternatively, first and second) dimensions.

[0129] exist Figure 9 In the example, modify the Figure 7 The architecture can include a further downsampling path at a lower resolution than the lower (e.g., half) resolution path.

[0130] like Figure 9As can be seen in FIG, some examples herein take as input 701 an image (e.g., a frame or a proper subset of a frame) represented by x' and a residual signal. In some examples, some portions (e.g., blocks) of the input 701 are spatially downsampled 702 in a lower (e.g., half) resolution path and processed using a series of residual blocks 704A, some portions (e.g., blocks) of the input 701 are spatially downsampled 902 in an even lower (e.g., quarter) resolution path and processed using a series of residual blocks 904, and some portions (e.g., blocks) of the input 701 (that are not downsampled) are processed in a full resolution path using a series of residual blocks 704B. In some examples, the output of the residual block 904 is upsampled at 906 (e.g., to increase the resolution of the downsampled 902 by a factor of two) and concatenated at 908 with the output of the series of residual blocks 704A at a lower (e.g., half) resolution. In some examples, the output (lower (e.g., half) resolution) from the stitching at 908 is provided to a fused scale operation 910, which converts the low-resolution (e.g., half) data into a predicted value for each sample location. In some examples, the predicted values from the fused scale operation 910 are upsampled at 706 and stitched at 708 with, for example, the output of a series of high-resolution residual blocks 704B and a convolution (e.g., conv2d k3 nS0) operation 712. In some examples, this stitched result (e.g., at full resolution) is provided to a fused scale operation 710, which converts the low-resolution and high-resolution data into a predicted value 714 for each sample location. In some examples, the predicted values are feature values (e.g., features for each of the red, green, and blue channels of the image). In some examples, different channels of the same block (e.g., a luma channel, a chroma channel, and another channel) are processed on different paths, for example, where a first channel (e.g., luma) of the block is processed on a full resolution path (e.g., by a series of residual blocks 704B), a second channel (e.g., chroma) of the same block is processed on a lower (e.g., 1 / 2) resolution path (e.g., by a series of residual blocks 704A), and a third channel (e.g., another channel) of the same block is processed on an even lower (e.g., 1 / 4) resolution path (e.g., by a series of residual blocks 904). In some examples, this allows for power and processing savings on each of the lower resolution paths, for example, compared to also performing processing of the second channel (e.g., chroma) of the same block at full resolution, and compared to also performing processing of the third channel (e.g., another channel) of the same block at full resolution (or 1 / 2 resolution).

[0131] exist Figure 9In the example in , the output of the quarter-resolution processed residual block is upsampled and fused with the output of the half-resolution processed residual block.

[0132] Figure 10 is a high-level architecture diagram 1000 for multi-scale (e.g., full-scale, half, and quarter resolution) processing using a machine learning (e.g., predictive) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples.

[0133] In an alternative example, such as Figure 10 As shown in the high-level architecture diagram of multi-scale processing, the output of the quarter-resolution processing residual block 904 is upsampled to full resolution at 1006 (e.g., Figure 9 ) and combined with the output of the full-resolution processed residual block.

[0134] like Figure 10 As can be seen in , some examples herein take as input 701 an image (e.g., a frame or a proper subset of a frame) denoted by x' and a residual signal. In some examples, some portions (e.g., blocks) of the input 701 are spatially downsampled 702 in a lower (e.g., half) resolution path and processed using a series of residual blocks 704A, some portions (e.g., blocks) of the input 701 are spatially downsampled 902 in an even lower (e.g., quarter) resolution path and processed using a series of residual blocks 904, and some portions (e.g., blocks) of the input 701 (that are not downsampled) are processed in a full resolution path using a series of residual blocks 704B. In some examples, the output of residual block 904 is upsampled at 1006 (e.g., to increase the resolution of downsampled 902 by a factor of four), the output of residual block 704A is upsampled at 706 (e.g., to increase the resolution of downsampled 702 by a factor of two), and both of those outputs are concatenated at 708 with, for example, the output of a series of high-resolution residual blocks 704B and convolution (e.g., conv2d k3 nS0) operations 712. In some examples, this concatenated result (e.g., at full resolution) is provided to a fused scale operation 710, which converts the low-resolution and high-resolution data into prediction values 714 for each sample location. In some examples, the prediction values are feature values (e.g., features for each of the red, green, and blue channels of the image).

[0135] In the above Figure 7 、 Figure 9 and Figure 10In some examples, the input is directly downsampled to a lower resolution. However, in some examples, progressive downsampling can be used. In some examples, this has the benefit of reducing complexity.

[0136] Figure 11 1100 is a high-level architecture diagram of progressive downsampling multi-scale (e.g., full-scale, half, and quarter resolution) processing using a machine learning (e.g., predictive) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples.

[0137] For example, Figure 11 (High-Level Architecture Diagram for Multi-Scale Processing) shows an example where half-resolution downsampling is computed from full-resolution processed data and quarter-resolution downsampling is a function of the half-resolution processed data.

[0138] Some of the above examples use residual blocks, scale-fusion operations, and spatial scaling operations, which are described in more detail below.

[0139] Residual Block

[0140] Figure 12 An example of a residual block is shown in In some examples, any residual block herein (eg, any of residual blocks 704A, 704B, 904, etc.) is an instance of residual block 1200.

[0141] Figure 12 1 is a diagram illustrating a residual block 1200 of a machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples. In some examples, the residual block 1200 takes as input a feature having one or more channels and processes the feature using a two-dimensional convolution (e.g., conv2d kK0 nC0) operation 1202, followed by an activation operation 1204, followed by a two-dimensional convolution (e.g., conv2d kK1 nC1) operation 1206. In some examples, the result is added to the output of another convolution (e.g., conv2d kK2 nC1) operation 1208. Figure 12 In

[15] , the conv2d kX nC operation represents a two-dimensional convolution with a spatial support of X×X samples and C channels as output. In some examples, when S is equal to 1, this is equivalent to the conv2d kX nC sS operation, and sS represents the stride of the convolution. In alternative examples, one or more of the two-dimensional convolution (conv2d) operations are replaced with operations with different dimensions, such as one-dimensional convolution, two-dimensional convolution, and / or three-dimensional convolution. One description of the conv2d kK0nC0 operation is as follows:

[0142]

[0143] Where star* is a valid 2D cross-correlation operator, N is the batch size, C represents the number of channels, H is the height of the input plane in terms of pixels or samples, and W is the width in terms of pixels or samples. This can include bias terms, for example, as in the reference Figure 7 discussed.

[0144] Figure 13 Shown is the input-output relationship of a rectified linear unit (ReLU) operation according to some examples. Figure 13 (Input-Output Relationship of a Rectified Linear Unit (ReLU) Operation) shows an example of an activation function, such as the activation function in multiple figures (e.g., Figure 12 1204 in activation function 1204). This example is often referred to as a rectified linear unit (or ReLU). In some examples, this operation is performed on each element of the input. In some examples, the ReLU operation can be fused with other operations (such as convolution operations).

[0145] Figure 14 The input-output relationship of the sigmoid operation according to some examples is shown. Figure 14 (Input-output relationship of sigmoid operation) shows another example of an activation function, such as the activation function in multiple figures (e.g., Figure 12 1204 in activation function). This example is often referred to as a sigmoid. In some examples, this operation is performed on each element of the input. Other example activation operations include parameterized rectified linear units.

[0146] Figure 15 An alternative example of a residual block is shown. Figure 15 is a diagram illustrating a residual block 1500 of a machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples. Figure 12 Compared to the residual block 1200 in , the residual block 1500 does not include the convolution (e.g., conv2d kK2 nC1) operation 1208.

[0147] Figure 16 Yet another example of a residual block is shown. Figure 16is a diagram illustrating a residual block 1600 of a machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples. In some examples, the residual block 1600 has the benefit of allowing parallel computation of a two-dimensional convolution (conv2d) operation 1602 (e.g., followed by activation 1604) and a two-dimensional convolution (conv2d) operation 1606. In some examples, this also has the benefit of having a smaller spatial extent than some of the above examples. In some examples, this has the benefit of reducing the row buffer requirements of the residual block. In some examples, the multiplication in the residual block 1600 is element-by-element multiplication. In some examples, the multiplication ("×") is element-by-element (e.g., point-by-point) multiplication, and the addition "+" is element-by-element (e.g., point-by-point) addition.

[0148] Fusion scale block

[0149] In some examples, channels from different scales are combined in a fusion scale block, e.g. Figure 7 、 Figure 9 and Figure 10 The fusion scale 710 (and Figure 11 fusion scale in ) and / or Figure 9 In one example, the fused scale block 1700 is implemented using a two-dimensional convolution (e.g., conv2d kK0 n1) operation 1704. In some examples, the operation 1704 generates a channel as output (e.g., and obtains its input from channel concatenation 1702), which corresponds to the fused channel. Figure 17 (using the fusion spatial resolution scale of the convolutional layer). In some examples, the splicing channel 1702 is any splicing channel operation herein, for example, Figure 7 and Figure 9 The splicing channel 708, Figure 9 The stitching channel 908 and / or Figure 11 The stitching channels shown (e.g., where the half-resolution processing path has a fused scale operation with the same number of output channels as input channels).

[0150] In another example, the fused scale block consists of one or more residual blocks. Figure 18 (Fusion Spatial Resolution Scale Using Residual Blocks) shows an example where the number of output channels is the same as the input channels.

[0151] Figure 18is a diagram illustrating a fusion scale block (using a residual block) of a machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples. In one example, the fusion scale block 1800 is implemented using a first two-dimensional convolution (e.g., conv2d kK0nC) operation 1804, which obtains its input from a channel splicing 1802, outputs to an activation function 1806 (e.g., ReLU), and the output of the activation function is provided as input to a second two-dimensional convolution (e.g., conv2d kK0 nC) operation 1808. In some examples, the splicing channel 1802 is any splicing channel operation herein, e.g., Figure 7 and Figure 9 The splicing channel 708, Figure 9 The stitching channel 908 and / or Figure 11 In some examples, 1802 provides input features (e.g., the number of elements). In some examples, the circled “∑” is a summation, for example, a summation of corresponding elements (e.g., element-by-element) (e.g., point-by-point).

[0152] In another example, the fusion scale block can use a combination of residual blocks and convolution blocks. Figure 19 (Fused spatial resolution scale using residual and convolutional blocks) shows an example with one output channel. Figure 19 is a diagram illustrating a fused scale block 1900 (using residual blocks and convolution blocks) of a machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples. In one example, the fused scale block 1900 is implemented using a first two-dimensional convolution (e.g., conv2d kK0 nC) operation 1904, which receives its input from a channel splicing 1902, outputs an activation function 1906 (e.g., ReLU), and the output of the activation function is provided as input to a second two-dimensional convolution (e.g., conv2d kK0 nC) operation 1908. In some examples, the output of the second two-dimensional convolution (e.g., conv2d kK0 nC) operation 1908 and the output of the spliced channel 1902 are used to generate a residual, and the residual is provided as input to a third two-dimensional convolution (e.g., conv2d kK0 n1) operation 1910.

[0153] In some examples, stitching channel 1902 is any stitching channel operation herein, e.g., Figure 7 and Figure 9 The splicing channel 708, Figure 9 The stitching channel 908 and / or Figure 11 Stitched channels shown.

[0154] Space expansion and contraction

[0155] Spatial scaling operations change the spatial resolution of the input tensor. In some examples, downsampling is an operation that reduces the spatial resolution of input features, while upsampling is an operation that increases the spatial resolution of input features. Some examples for implementing spatial scaling are described below.

[0156] Figure 20 Downsampling 2000 using strided convolution according to some examples is shown. Figure 20 An example of downsampling using stride is shown. Figure 20 In , the convolution operation takes as input 2002 a feature tensor with two channels. The spatial dimension of the features is 4×4 samples (shown as white squares in 2002), and the operation outputs 2004 one channel, where the downsampling factor is 2. In one example, downsampling is achieved by first padding the input tensor with zero values, as in Figure 20 As shown in the input 2002 in . In some examples, the stride is set to 2 in each spatial dimension, which determines the number of output samples. Figure 20 As shown, in this example, the kernel has dimensions of three in both spatial dimensions and two in the channel dimension. Figure 20 In FIG, four shaded and / or cross-hatched samples are used to illustrate the kernel support region in the input. The output 2004 sample values have corresponding shading and / or cross-hatching. The convolution operation itself can be expressed as a conv2dk3n1s2 operation, or equivalently a conv2dk3n1s2p1 operation, where p1 represents zero padding.

[0157] Each output position (e.g., output channel) corresponds to the addition of a bias value and 12 corresponding shaded and / or cross-lined samples in the input multiplied by the sum of 12 kernel weights. When the output has more than one channel (where each output channel corresponds to one kernel), multiple convolution kernels are used.

[0158] exist Figure 20 In the example above, the convolution operation uses all channels in the input. In other examples, it may be desirable to group the different input channels into channel groups and restrict the convolution operation to a single group. In this case, the number of groups (or alternatively, the group size) is specified for the convolution layer. An example of using channel groups is a downsampler that operates independently on each input channel. This is done in Figure 21 (Downsampling using strided convolution and input channel grouping).

[0159] Figure 21 Downsampling 2100 using strided convolution and input channel grouping according to some examples is shown. Figure 21In , there are as many channel groups as input channels. Each kernel is supported to span one input channel, which corresponds to a channel group size of 1.

[0160] In another example, it is desirable to have a channel group size greater than 1, for example, a convolution operation then uses a kernel that spans more than one channel. In some examples, a convolution operator with a group size of G is represented as a conv2d kK0 nC0 sSgsG operation, where G is assumed to be equal to the number of input channels by default. In some examples, using channel groups reduces the complexity of the two-dimensional convolution (conv2d) operation because each convolution kernel operates on a smaller number of input channels. Thus, while a conv2d k3 n2 operation with two input channels performs 2×3×3 operations per output sample, a conv2d k3 n2 gs1 operation with two input channels performs 1×3×3 operations per output sample.

[0161] In some examples, the channel groups are not of the same size. In the same or other examples, the number of channels output by each channel group need not be the same. Figure 22 (Example Channel Grouping for Convolutional Layers) shows example channel groups for conv2d operations. Figure 22 Three different channel groupings of a convolutional layer according to some examples are shown. For example, a many-to-many channel grouping 2202 (shown as four channel groups 0-3 of input each having four channels, and each output group having three channels), a many-to-one channel grouping 2204 (shown as four channel groups 0-3 of input each having four channels, and each output group having one channel), and a one-to-many channel grouping 2206 (shown as four channel groups 0-3 of input each having one channel, and each output group having four channels).

[0162] Figure 23 Upsampling by pixel shuffling is shown according to some examples. Figure 23 (Upsampling using pixel shuffling) shows an example of pixel shuffling. Figure 23 As shown, the input 2302 consists of a tensor with four channels and six samples in horizontal and vertical dimensions (shown as four different 6×6 2D matrices). The output 2304 of the pixel shuffle operation corresponds to the interleaving of the four input channels to create a tensor with one channel and 12 samples in horizontal and vertical dimensions (shown as a 12×12 2D matrix). In some examples, the pixel shuffle (2) operation rearranges the samples to create a single channel that is twice the spatial size of the input channels (e.g., shown as 6×6 input channels and a single 12×12 output channel).

[0163] Figure 23(Upsampling using pixel shuffling) shows an example of pixel shuffling. Figure 23 As shown, the input 2302 consists of a tensor with four channels and six samples in horizontal and vertical dimensions (shown as four different 6×6 2D matrices). The output 2304 of the pixel shuffle operation corresponds to the interleaving of the four input channels to create a tensor with one channel and 12 samples in horizontal and vertical dimensions (shown as a 12×12 2D matrix). In some examples, the pixel shuffle (2) operation rearranges the samples to create a single channel that is twice the spatial size of the input channels (e.g., shown as 6×6 input channels and a single 12×12 output channel).

[0164] In some examples of upsampling using pixel shuffling, a two-dimensional convolution (conv2d) operation is used to create the input to the pixel shuffling operation. For example, the operation can be conv2dkK0 cC1 gG, where C1 is equal to four times the number of input channels and G is equal to the number of input channels. In some examples, the kernel weights used by the conv2d operation can be determined using a training algorithm. Or, alternatively, corresponding to an upsampling algorithm such as, but not limited to, nearest neighbor interpolation, bilinear interpolation, and / or bicubic interpolation. Certain upsampling algorithms can have the benefit of lower complexity. Alternatively, the learned weights can better preserve information.

[0165] While channel grouping reduces complexity, an alternative approach for achieving reduced complexity in the conv2d operation is to reduce the spatial extent of the kernel. Some examples use convolution kernels with diamond, horizontal, vertical, or plus-sign shapes, such as Figure 24 shown. Figure 24 Twelve different convolution kernels are shown with diamond, horizontal, vertical, and plus-sign spatial extent shapes according to some examples.

[0166] In some examples, a diamond shape is used, represented as a capital D in the "conv2dD" operation.

[0167] In some examples, the kernel in the conv2d operation can be asymmetric about co-located samples in the input.

[0168] Additional Examples

[0169] Figure 25 (an example with a group size of 2 at half resolution) shows an example of a method using full-resolution and half-resolution processing paths. Figure 25 shows a full-resolution and half-resolution processing path 2500 with a group size of 2 in the half-resolution path according to some examples. Figure 25 shows an image represented as x' and residual data as input. The full-resolution path takes the input and applies a conv2d operation, followed by a batch normalization operation. The output of batch normalization is provided as input to a sequence of four residual blocks using a diamond shape for the conv2d operation. The half-resolution path takes the input and downsamples it (using strided convolution), followed by a batch normalization operation. The output of batch normalization is provided as input to a sequence of four residual blocks using a channel group size of 2 for the conv2d operation. Two of the residual blocks use a 3×1 kernel for the conv2d operation; the other two of the residual blocks use a 1×3 kernel for the conv2d operation. The output of the fourth residual block in the half-resolution processing path is the input to a convolution operation with a 1×1 kernel and a group size of 2. This convolution operation outputs one channel for each channel group, which has the benefit of reducing data size. The output of this convolution layer is fed into an upsampling operation with a pixel shuffle operation. In some examples, the pixel shuffle operation is preceded by a conv2d operation that outputs four times the number of channels as the input channels, for example, a conv2d operation with a group size of 1 results in 4 channels being output for each input channel. The output of the upsampling operation is concatenated with one full-resolution channel and input to a residual block with a 1×1 convolution and a group size of 4. This is followed by a convolution with a 1×1 spatial extent that outputs one channel.

[0170] Figure 26 FIGURE 1 illustrates a batch normalization operation according to some examples. In some examples, the batch normalization operation applies a series of multiplication and addition operations to each sample in a tensor. In one example, as Figure 26 The operation can be expressed as shown, for example, where bn0, bn1, bn2, and bn3 are the parameters of the batch normalization operation, input i is the i-th element of the input tensor, and out i is the i-th element of the output tensor. In some examples, the parameters (e.g., bn0, bn1, bn2, bn3) are different for different channels.

[0171] In some examples, the batch normalization operation can be combined with other operations (e.g., convolution).

[0172] The example in Figure 25 has several benefits. First, the reduced spatial extent in the high-resolution processing path reduces computational complexity. Second, the use of channel groups allows for parallel computation.

[0173] FIG27 shows a full-resolution and half-resolution processing path 2700 according to some examples, where some (e.g., four) half-resolution residual blocks use a group size (e.g., 6) and the remaining (half-resolution residual blocks use a group size of 8. FIG27 (an example with group sizes 6 and 8 at half-resolution) shows another example of a method using full-resolution and half-resolution processing paths. This example uses different group sizes in the half-resolution processing path. As can be seen in FIG27 , some (e.g., four) half-resolution residual blocks use a group size of 6, while the remaining half-resolution residual blocks use a group size of 8. This has the benefit of improving the accuracy of the prediction at the expense of increased network complexity.

[0174] FIG28 shows a full-resolution and half-resolution processing path 2800 according to some examples, wherein a single channel group is provided in the half-resolution path. FIG28 (an example with a single channel group at half-resolution) shows another example of a method using full-resolution and half-resolution processing paths. This example uses a single group in the half-resolution processing path. As can be seen in FIG28 , in some examples, all half-resolution blocks are included in the same group. This has the benefit of further improving the accuracy of the prediction at the expense of, for example, further increasing network complexity.

[0175] The examples herein may be located where desired, for example, within a prediction loop of a video codec or outside the prediction loop as a post-processor. In one example, one or more examples herein are included as loop filters of an encoder and / or decoder.

[0176] Figure 29 is a diagram illustrating a video code processing 2900 according to some examples that includes a machine learning (e.g., prediction) model 112 (e.g., a multi-scale convolutional neural network (MSCNN)) and can switch between the output of the machine learning model and the output of a constrained directional enhancement filter (CDEF) 426.

[0177] and Figure 4 In contrast, video code processing 2900 includes a first instance 422A of loop filter 422 and a second instance 422B of loop filter 422. In some examples, ML model 112 is included to process the output from summation 2906, e.g., to produce better quality pixel values (e.g., as frames of an image or video). Figure 29(A codec that can switch between the proposed method and / or the ML model and the deblocking / constrained directional enhancement filter (CDEF) herein) shows an example in which the ML model 112 (e.g., MSCNN) obtains the output of the inverse transform operation 418 (e.g., the image and / or residual discussed herein). As shown in the loop filter 422B, the deblocking operation 424 also takes the output of the inverse transform operation 418 (e.g., the image and / or residual discussed herein) as input, and the output of the deblocking operation 424 is provided as input to the constrained directional enhancement filter (CDEF) 426. In some examples, (e.g., only) one of the outputs of the ML model 112 or the CDEF is selected (e.g., by a switch 2908) and provided as input to the cross-component sample offset (CCSO) 428, super-resolution 2902, and loop recovery 2904 operations. In some examples, one of the outputs of the ML model 112 or the CDEF is selected (e.g., via switch 2908) based on ML model performance, e.g., by generating outputs of both the ML model 112 and the CDEF and selecting the one that is more efficient for code processing (e.g., lowest cost metric). In one or another example, a line buffer is shared between the ML model 112 and one or both of the deblocking and CDEF operations. In another or additional example, the selection is determined by information received in the bitstream.

[0178] Figure 30 is a diagram illustrating a video code processing 3000 according to some examples that includes replacing the deblocking and constrained directional enhancement filter (CDEF) 426 with a machine learning (e.g., prediction) model 112 (e.g., a multi-scale convolutional neural network (MSCNN)). Figure 30 (The proposed method herein replaces deblocking and constrained directional enhancement filter (CDEF)) Another example is shown where the ML model 112 is located within the prediction loop of the video codec (e.g., loop filter 422). In the example, the ML model replaces the deblocking and constrained directional enhancement filter (CDEF) 426.

[0179] Figure 31 is a diagram showing a video code processing 3100 according to some examples, which includes a machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)) that takes as input the output of a constrained directional enhancement filter (CDEF). Figure 31Yet another example is shown where the ML model 112 takes the output of the CDEF operation 426 as input, for example, and the ML model 112 sends its output to a cross-component sample offset (CCSO) 428 .

[0180] Different configurations of the proposed methods and / or ML models (e.g., and other loop operations) herein have benefits. In one example, the proposed methods and / or ML models replace super-resolution operations. In another example, the codec switches between the proposed methods (and / or ML models) and super-resolution operations. In another example, the proposed methods and / or ML models replace loop recovery operations. And in yet another example, the codec switches between the proposed methods and / or ML models and loop recovery operations.

[0181] In yet another example, the output of the proposed method and / or ML model is fed into a deblocking operation. This has the benefit of attenuating block boundaries.

[0182] The proposed method and / or ML model can also be configured as a post-processor (e.g., a post-processing operation). Figure 32 An example is shown in . Figure 32 32 is a diagram illustrating a video code process 3200 including a machine learning (e.g., prediction) model 112 (e.g., a multi-scale convolutional neural network (MSCNN)) implemented as a post-processor 140 according to some examples. Figure 32 As can be seen, the proposed method and / or ML model takes as input the output image of the video decoder 602 (e.g., from the decoded picture buffer 624) and provides an enhanced image as output 310 (or 314). In some examples, the method and / or ML model also receives information from the bitstream and entropy decoder 602. Note that in some examples, the information in the bitstream does not require entropy decoding and is provided directly to the method. In some examples, the post-processor has multiple functional blocks and the ML method is not the first block, for example, where the input to the post-processor is the output of the video decoder, which may be modified by one or more post-processing operations before being input to the ML model.

[0183] enter

[0184] Some examples herein have included images and residual data as inputs. These examples are not intended to represent limitations on the inputs, and some examples obtain other data. For example, the method may depend on luma sample values, chroma sample values, dequantized inverse transform coefficients, slice type values, prediction information, chroma format information, relative positions of luma and chroma sample information, luma quantization parameters, chroma quantization parameters, temporal layer values, and / or other information. The data may correspond to a current processing position in an image, a previous processing position in an image, or a processing position in another image. The data may be scaled, cropped, and / or otherwise processed before being input to the method.

[0185] In an example, parameters for operations within a method can be selected based on quantization parameters. For example, a conv2d operation includes kernel parameters and bias parameters for computing the output of the convolution operation. Alternatively, a batch normalization operation includes scaling parameters and offset parameters for computing the output of the batch normalization operation. Such parameters can be referred to as method parameters. In one example, a first set of method parameters is associated with a first range of quantization values, and a second set of method parameters is associated with a second range of quantization values. In another example, the selection of method parameters is determined by both the slice type and the quantization parameter. For example, a first set of method parameters is associated with a first range of quantization values and a first slice type, a second set of method parameters is associated with a second range of quantization values and a first slice type, and a third set of method parameters is associated with a first range of quantization values and a second slice type. For example, a slice is a region of a frame within a video encoded (e.g., AVC or HEVC) that is encoded only with respect to that region rather than the entire frame. Other examples associated with method parameter sets may include, but are not limited to, prediction type values, temporal layer values, and / or block-level indicator values.

[0186] Signaling

[0187] The method may be controlled by information in the bitstream. In a first example, the method is enabled or disabled by signaling a flag from the encoder, receiving a flag at the decoder, and / or receiving a flag at the post-processor.

[0188] Figure 33 Shown is a syntax structure 3300 for signaling a flag in a sequence header according to some examples. Figure 33 A syntax structure 3300 for signaling a flag in a sequence header is shown. Without loss of generality, certain syntax and semantics from the AV1 specification are used herein, but other syntax and semantics (e.g., from other standards) may be used.

[0189] The semantics of structure 3300 include:

[0190] enable_nn_operation_seq equal to 1 specifies that the neural network filtering operation can be enabled.

[0191] enable_nn_operation_seq equal to 0 specifies that the neural network filter operation is disabled.

[0192] In some examples, enable_nn_operation_seq may be equal to 1, but then the proposed method and / or ML model may be disabled on a frame and / or block basis.

[0193] Additional parameters of the proposed ML model can be indicated in the syntax structure.

[0194] Figure 34 Shown is a syntax structure 3400 for signaling model parameters in a sequence header according to some examples. Figure 34 The syntax structure 3400 in FIG. 3400 shows an example of including parameters in the uncompressed_header syntax structure. In the example, nn_operation_params() represents a syntax structure containing model parameters.

[0195] In some examples, when the proposed method and / or ML model is enabled at the sequence level, it can be further enabled or disabled at the block level, frame level, tile level, or slice level. Figure 35 A grammatical structure 3500 is shown for enabling (or disabling) a machine learning (e.g., predictive) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples.

[0196] Figure 35 An example of a nn_operation_params syntax structure 3500 is shown for enabling (or disabling) the proposed method and / or ML model. The semantics of the syntax structure 3500 include:

[0197] enable_nn_operation equal to 1 specifies that the proposed method and / or ML model can be enabled for the picture.

[0198] enable_nn_operation equal to 0 specifies that the proposed method and / or ML model is disabled for the image.

[0199] The model_selector syntax structure can be used to indicate the method and / or model parameters to be used by the ML model. Figure 36 An example is shown in . Figure 36A grammatical structure 3600 for selecting a model parameter set according to some examples is shown. Although the number of models shown in some figures is four, one of ordinary skill in the art will appreciate that a single model or any number of models may be utilized.

[0200] In this example, multiple model parameter sets are defined, and the model_selector syntax structure 3600 indicates which of the multiple model parameter sets is to be used. For example, when four candidate model parameter sets are available, a two-bit index identifying the selected model parameter set may be signaled and / or received, such as Figure 36 shown.

[0201] The semantics of the grammatical structure 3600 include:

[0202] model_idc equal to 0 indicates that the first model parameter set is selected for the frame. model_idc equal to 1 indicates that the second model parameter set is selected for the frame. model_idc equal to 2 indicates that the third model parameter set is selected for the frame. model_idc equal to 3 indicates that the fourth model parameter set is selected for the frame.

[0203] Figure 37 A second example of the model_selector syntax structure is shown in . Figure 37 A grammatical structure 3700 for selecting one or more sets of model parameters is shown according to some examples.

[0204] In this example, an ordered list of model parameter sets is constructed, and a bit mask is used to indicate one or more model sets from the list. Each bit in the mask corresponds to a model parameter set at a position in the ordered list. In some examples, the number of sets to be selected is predefined. In these cases, once the required number of model sets has been selected, there is no need to indicate the remainder of the mask. Figure 37 The syntax structure for indicating three model parameter sets from a set of 12 candidates using a truncated mask is shown. Note that despite the presence of 12 models, the mask length never exceeds 11 bits, as the last bit can be inferred based on how many choices have been indicated previously.

[0205] The semantics of the grammatical structure 3700 include:

[0206] ModelSelected[model_idx] equal to 1 indicates that the model_idx model parameter set is selected.

[0207] ModelSelected[model_idx] equal to 0 indicates that the model_idx model parameter set is not selected.

[0208] In additional examples, the correspondence between list positions and model parameter sets is predefined. In other examples, the correspondence between list positions and model parameter sets can be derived. In one example, frequently used model parameter sets are assigned to earlier positions in the list.

[0209] Figure 38 An example of the nn_operation_scale syntax structure is shown in . Figure 38 A syntax structure 3800 for indicating a scale parameter according to some examples is shown. In the example, the value of NNScale is indicated and takes one of three values (e.g., 1.00, 0.75, 0.50). This value is used to scale the output of the proposed method and / or ML model by multiplying the output of the method and / or ML model by the NNScale value.

[0210] The semantics of the grammatical structure 3800 include:

[0211] nn_operation_scale_indicator0 equal to 1 specifies that nn_operation_scale_indicator1 is present in the bitstream.

[0212] nn_operation_scale_indicator1 equal to 1 specifies the NNScale parameter equal to 0.75. nn_operation_scale_indicator1 equal to 0 specifies the NNScale parameter equal to 0.50.

[0213] The scaling values can be specified for different channels in the method. For example, the scaling value can be applied to each channel before the channel splicing operation. Alternatively, the scaling factor can be applied to each channel before the fusion layer operation. Figure 39 An example syntax structure for indicating a scaling factor is shown in . Figure 39 A syntax structure 3900 is shown for indicating scale parameters for four output channels of a machine learning (e.g., prediction) model (e.g., a multi-scale convolutional neural network (MSCNN)) according to some examples. Figure 39 In the example, the model has four output channels. The corresponding four scaling values are stored in the array NNScale[], and each channel c is modified by the NNScale[c] value.

[0214] The example nn_operation_scale syntax structure shows the indication of one or more scaling values. However, it should be understood that other values can be indicated. For example, a bias value can be indicated. In addition, while the example contemplates the indication of values on a frame basis, other granularities are possible. For example, the nn_operation_scale syntax structure can indicate values at the block level, superblock level, tile level, slice level, or sequence level.

[0215] As described above, a model parameter set can be associated with a quantization parameter and a slice type value. In an example, multiple model parameter sets are associated with the quantization parameter and the slice type value. A model selector in the bitstream then indicates which model parameter set to use from the multiple sets.

[0216] Figure 40 An example assignment 4000 between model parameter sets, quantization parameter (QP) values, and picture types is shown according to some examples. Figure 40 A case is shown in which four model parameter sets are assigned to six quantization parameter (QP) ranges and two slice type values (e.g., intra picture values and inter picture values). Although there are a total of 48 model parameter sets, only four model parameter sets are available for selection for each combination of QP and slice type values. Therefore, a 2-bit indicator (as shown in Table 4) can indicate the model to be selected.

[0217] In another example, the selectable model parameter sets are indicated in a grammatical structure.In one example, the list of available models may be indicated by a count of available models followed by the corresponding model identifier. Figure 41 A grammatical structure 4100 is shown for indicating a set of model parameters available for selection according to some examples. In some examples, the grammatical structure 4100 may be used to indicate Figure 40 The availability is shown because a special model_identifier syntax element can be used to identify up to a threshold (e.g., 64) unique models for each QP parameter and slice type value.

[0218] The semantics of the grammatical structure 4100 include:

[0219] model_available_count_minus1 plus 1 defines the number of available model parameter sets.

[0220] Model_identifier identifies the set of model parameters to be assigned to index model_available_idx in the list of available models.

[0221] As mentioned above, the proposed method and / or ML model can be enabled or disabled at the block level. Figure 42A syntax structure 4200 for indicating block-level control according to some examples is shown. In one example, Figure 42 The presence of block-level controls is indicated in the nn_operation_params syntax structure 4200 shown.

[0222] The semantics of the grammatical structure 4200 include:

[0223] nn_operation_block_control_enable equal to 0 specifies not to enable or disable the method on a block basis.

[0224] nn_operation_block_control_enable equal to 1 specifies that the method can be enabled or disabled on a block basis.

[0225] nn_operation_block_size_idc equal to 0 indicates that the method is controlled at a 16×16 block granularity.

[0226] nn_operation_block_size_idc equal to 1 indicates that the method is controlled at a 32×32 granularity.

[0227] nn_operation_block_size_idc equal to 2 indicates that the method is controlled at a 64×64 granularity.

[0228] nn_operation_block_size_idc equal to 3 indicates that the method is controlled at a 128×128 granularity.

[0229] Figures 43 to 44 Shown are syntax structures for indicating block-level controls 4300 , 4400 according to some examples.

[0230] In these tables, an array NNOperationUnitSize[] = {16, 32, 64, 128} is defined to map the block size indicator to the block size.

[0231] Figure 44 The semantics include:

[0232] ApplyNNOperationToUnit[unitRow][unitCol] equal to 1 specifies that the method applies to the block located at unitRow, unitCol in the picture.

[0233] ApplyNNOperationToUnit[unitRow][unitCol] equal to 0 specifies that the method is not applied to the block located at unitRow, unitCol in the picture.

[0234] Network Assembly

[0235] Some examples consider the selection of a set of model parameters for use in the method. In some examples, this has the benefit of improving code processing efficiency because only an indicator of the set is required, and it is not necessary to directly indicate all model parameters. However, as the number of model parameter sets increases, indicating the selected model may become cumbersome. Some examples herein use a network assembly method to signal this selection, which has the benefit of improving code processing efficiency in these cases.

[0236] In some examples, a network assembly method selects subsets of model parameters for different operations in the method. For example, Figure 9 The illustrated method and / or architecture diagram includes two sub-models: a high-resolution processing path and a half-resolution processing path. Using a network assembly method, model parameters for the high-resolution processing path are selected from a high-resolution processing path parameter set. Furthermore, model parameters for the low-resolution processing path are selected from a low-resolution processing path parameter set. Without loss of generality, the selection of each subset can be indicated using one of the previous examples for indicating a model parameter set.

[0237] In the second example of the network assembly method, the scaling factors of the outputs of the sub-models are indicated. For example, referring again to Figure 9 , model parameters for the high-resolution processing path are selected from the high-resolution processing path parameter set. And model parameters for the low-resolution processing path are selected from the low-resolution processing path parameter set. In addition, a scaling factor is indicated, and the scaling factor is applied to the channels output by the two processing paths before the splice channel operation. Alternatively, the scaling factor is applied before the fusion scale operation. Without loss of generality, the selection and scaling factor can be indicated using one of the previous examples.

[0238] Figure 45 A network assembly method 4500 according to some examples is shown. Figure 45 In , s0 and s1 represent the first and second processing paths, respectively. And NNi represents the i-th model parameter set, respectively. Thus, s0_NN0 indicates the first model parameter set of the first processing path. The selector_s0 and selector_s1 operations select one or more of the outputs from the model parameter set and provide the outputs to the channel splicing operation. Figure 45As can be seen in Figure 1, selector_s0 selects output from one of the four model parameter sets of the first processing path; selector_s1 selects output from three of the 12 model parameter sets of the second processing path. These selected channels are input to the concatenate channels operation, followed by a conv2d operation. In this example, the parameters of the conv2d operation are fixed. However, in other examples, the parameters can depend on the selected model parameters indicated in the bitstream or selected from the model parameter set.

[0239] exist Figure 45 In some implementations of , the outputs of sub-models not selected by the selector are not computed. In addition, the concatenation channel and conv2d operations can be replaced by other operations.

[0240] Figure 46 4 is a flowchart illustrating operation 4600 of a method for using a multi-scale machine learning model according to some examples. Some or all of operation 4600 (or other processes described herein, or variations and / or combinations thereof) are performed under the control of one or more computer systems configured with executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executed together on one or more processors, by hardware, or a combination thereof. The code is stored on a computer-readable storage medium, for example, in the form of a computer program including instructions executable by one or more processors. The computer-readable storage medium is non-transitory. In some examples, one or more (or all) of operations 4600 are performed by devices of other figures (e.g., device 5100) and / or content distribution system 102 (e.g., implemented in a provider network).

[0241] Operation 4600 includes, at block 4602, receiving a video at a content distribution service. Operation 4600 includes, at block 4604, generating, by a multi-scale machine learning model, a prediction based on an input frame of the video. Operation 4600 includes, at block 4606, encoding, by the content distribution service, the input frame of the video based on the prediction to generate an encoded frame. Operation 4600 includes, at block 4608, transmitting the encoded frame from the content distribution service to a viewer device.

[0242] Figure 474 is a flowchart illustrating the operations of a method for generating a modified version of a frame based on a first feature set generated by a machine learning model and an upsampled second feature set according to some examples. Some or all of operation 4700 (or other processes described herein, or variations and / or combinations thereof) are performed under the control of one or more computer systems configured with executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executed together on one or more processors, by hardware, or a combination thereof. The code is stored on a computer-readable storage medium, for example, in the form of a computer program including instructions executable by one or more processors. The computer-readable storage medium is non-transitory. In some examples, one or more (or all) of operations 4700 are performed by devices (e.g., device 5100) and / or content distribution system 102 (e.g., implemented in a provider network) of other figures.

[0243] Operation 4700 includes, at block 4702, performing video coding processing on a frame of a video, the video coding processing generating first pixel values and first residuals for a block of the frame. Operation 4700 further includes, at block 4704, generating a first feature set by a machine learning model based on the first pixel values and first residuals of the block at a first resolution. Operation 4700 further includes, at block 4706, generating a second feature set by the machine learning model based on the second pixel values and second residuals of the block at a second, lower resolution. Operation 4700 further includes, at block 4708, upsampling the second feature set to the first resolution to generate an upsampled second feature set. Operation 4700 further includes, at block 4710, generating a modified version of the frame based on the first feature set and the upsampled second feature set. Operation 4700 further includes, at block 4712, transmitting the modified version of the frame to a frame buffer or a display device.

[0244] Exemplary environments, systems, etc., in which the foregoing may be used are described in detail below.

[0245] At least some examples of the disclosed technology can be described in terms of:

[0246] Clause 1. A computer-implemented method comprising:

[0247] receiving the video at a content distribution service;

[0248] generating, by a multi-scale machine learning model, predictions based on input frames of the video;

[0249] performing, by the content distribution service, encoding of the input frame of the video based on the prediction to generate an encoded frame; and

[0250] The encoded frames are transmitted from the content distribution service to a viewer device.

[0251] Clause 2. A computer-implemented method as described in Clause 1, wherein generating the prediction by the multi-scale machine learning model is performed within a prediction loop of a video codec.

[0252] Clause 3. A computer-implemented method as described in Clause 1, wherein generating the prediction by the multi-scale machine learning model is performed within a post-processor service after the decoder.

[0253] Clause 4. A computer-implemented method as described in Clause 1, wherein the prediction generated by the multi-scale machine learning model is based on the input frame and the residual value.

[0254] Clause 5. The computer-implemented method of clause 4, further comprising:

[0255] generating quantized coefficients of the input frame;

[0256] generating inverse quantized coefficients of the quantized coefficients; and

[0257] The residual value is determined based on the inverse quantized coefficients.

[0258] Clause 6. The computer-implemented method of Clause 1, wherein deblocking and constrained directional enhancement filters of the predictive replacement video codec are generated.

[0259] Clause 7. A computer-implemented method as described in Clause 1, wherein the prediction generated by the multi-scale machine learning model is based on an inverse transform of the input frame.

[0260] Clause 8. A computer-implemented method comprising:

[0261] receiving the video at a content distribution service;

[0262] performing encoding, by the content distribution service, on frames of the video, the encoding converting the frames from a pixel domain to a transform (e.g., frequency) domain and back to the pixel domain to generate first pixel values and first residuals for blocks of the frames at a first resolution;

[0263] generating, by a machine learning model of the content distribution service, a first feature set based on inputs of the first pixel value and the first residual of the block at the first resolution;

[0264] generating, by the machine learning model of the content distribution service, a second feature set for input of a second pixel value and a second residual of the block at a second lower resolution;

[0265] upsampling the second feature set to the first resolution to generate an upsampled second feature set;

[0266] generating a modified version of the frame based on the first feature set and the upsampled second feature set; and

[0267] The modified version of the frame is transmitted to a frame buffer or from the content distribution service to a viewer device.

[0268] Clause 9. The computer-implemented method of clause 8, further comprising:

[0269] generating, by the machine learning model of the content distribution service, a third feature set for input of third pixel values and third residuals of the block at a third resolution lower than the second lower resolution; and

[0270] upsampling the third feature set to the first resolution to generate an upsampled third feature set,

[0271] Wherein generating the modified version of the frame is based on the first feature set, the upsampled second feature set, and the upsampled third feature set.

[0272] Clause 10. The computer-implemented method of Clause 8, wherein generating the first feature set, generating the second feature set, and generating the modified version of the frame occur within a loop filter of an encoder.

[0273] Clause 11. A computer-implemented method comprising:

[0274] performing video coding processing on a frame of a video, the video coding processing generating first pixel values and first residuals for a block of the frame;

[0275] Generating a first feature set by a machine learning model based on the first pixel value and the first residual of the block at a first resolution;

[0276] generating, by the machine learning model, a second feature set for input of a second pixel value and a second residual of the block at a second lower resolution;

[0277] upsampling the second feature set to the first resolution to generate an upsampled second feature set;

[0278] generating a modified version of the frame based on the first feature set and the upsampled second feature set; and

[0279] The modified version of the frame is transmitted to a frame buffer or a display device.

[0280] Clause 12. The computer-implemented method of clause 11, further comprising:

[0281] generating, by the machine learning model, a third feature set for input of third pixel values and third residuals of the block at a third resolution lower than the second lower resolution; and

[0282] upsampling the third feature set to the first resolution to generate an upsampled third feature set,

[0283] Wherein generating the modified version of the frame is based on the first feature set, the upsampled second feature set, and the upsampled third feature set.

[0284] Clause 13. The computer-implemented method of Clause 11, wherein generating the first feature set, generating the second feature set, and generating the modified version of the frame occur within a loop filter of an encoder.

[0285] Clause 14. The computer-implemented method of Clause 11, wherein a viewer device comprises a decoder and the display device, and generating the first feature set, generating the second feature set, and generating the modified version of the frame occur within a loop filter of the decoder.

[0286] Clause 15. The computer-implemented method of Clause 14, further comprising:

[0287] determining an indication of a subset of blocks of a frame to be processed by a machine learning model of the decoder; and

[0288] The indication is sent to the decoder to cause the decoder to process the subset of blocks of the frame through the machine learning model of the decoder.

[0289] Clause 16. The computer-implemented method of Clause 11, wherein generating the first feature set, generating the second feature set, and generating the modified version of the frame occur in a post-processor separate from any encoder and any decoder.

[0290] Clause 17. The computer-implemented method of Clause 11, further comprising: downsampling the block from the first resolution to the second lower resolution before generating the second feature set.

[0291] Clause 18. The computer-implemented method of Clause 17, wherein the downsampling comprises performing a strided convolution on the block at the first resolution.

[0292] Clause 19. The computer-implemented method of Clause 11, wherein the upsampling comprises interleaving multiple channels into one channel.

[0293] Clause 20. The computer-implemented method of Clause 11, wherein generating the modified version of the frame comprises performing a cross-component sample offset operation.

[0294] Clause 21. The computer-implemented method of Clause 11, further comprising selecting one of the modified version of the block and another version of the block as input to a cross-component sample offset operation.

[0295] Clause 22. A non-transitory computer-readable medium storing code that, when executed by an apparatus, causes the apparatus to perform a method comprising:

[0296] performing video coding processing on a frame of a video, the video coding processing generating first pixel values and first residuals for a block of the frame;

[0297] Generating a first feature set by a machine learning model based on the first pixel value and the first residual of the block at a first resolution;

[0298] generating, by the machine learning model, a second feature set for input of a second pixel value and a second residual of the block at a second lower resolution;

[0299] upsampling the second feature set to the first resolution to generate an upsampled second feature set;

[0300] generating a modified version of the frame based on the first feature set and the upsampled second feature set; and

[0301] The modified version of the frame is transmitted to a frame buffer or a display device.

[0302] Clause 23. The non-transitory computer-readable medium of Clause 22, wherein the method further comprises:

[0303] generating, by the machine learning model, a third feature set for input of third pixel values and third residuals of the block at a third resolution lower than the second lower resolution; and

[0304] upsampling the third feature set to the first resolution to generate an upsampled third feature set,

[0305] Wherein generating the modified version of the frame is based on the first feature set, the upsampled second feature set, and the upsampled third feature set.

[0306] Clause 24. The non-transitory computer-readable medium of Clause 22, wherein generating the first feature set, generating the second feature set, and generating the modified version of the frame occur within a loop filter of an encoder.

[0307] Clause 25. The non-transitory computer-readable medium of Clause 22, wherein the device comprises a decoder and the display device, and generating the first feature set, generating the second feature set, and generating the modified version of the frame occur within a loop filter of the decoder.

[0308] Clause 26. The non-transitory computer-readable medium of Clause 22, wherein generating the first feature set, generating the second feature set, and generating the modified version of the frame occur in a post-processor separate from any encoder of the device and any decoder of the device.

[0309] Clause 27. The non-transitory computer-readable medium of Clause 22, wherein generating the modified version of the frame comprises performing a cross-component sample offset operation.

[0310] Clause 28. A computer-implemented method comprising:

[0311] receiving the video at a content distribution service;

[0312] downsampling frames of the video from a first resolution to a second, lower resolution in vertical and horizontal directions;

[0313] performing, by the content distribution service, encoding on the frame of the video, the encoding converting the frame from a pixel domain to a transform (e.g., frequency) domain and back to the pixel domain to generate a result for the frame at the second lower resolution;

[0314] upsampling the result in the vertical direction and the horizontal direction to a resolution higher than the second lower resolution to generate an upsampled result;

[0315] performing entropy encoding of the frame based at least on the upsampled result to generate a coded frame; and

[0316] The transcoded frames are transmitted from the content distribution service to a decoder (eg, of a viewer device).

[0317] Clause 29. The computer-implemented method of Clause 28, further comprising:

[0318] generating a reference frame for the frame based on the upsampled result;

[0319] storing the reference frame in a frame buffer; and

[0320] Corresponding decoded motion vectors for the frame at the second lower resolution are stored in the frame buffer.

[0321] Clause 30. The computer-implemented method of Clause 28, wherein the upsampling comprises generating the upsampled result by a machine learning model. In some clauses, the upsampled result is generated by performing a machine learning process on an output from the upsampling, and the output of the machine learning model is the upsampled result generated by the machine learning model.

[0322] Clause 31. A computer-implemented method comprising:

[0323] receiving a transcoded frame of a video;

[0324] performing video transcoding on the transcoded frames of the video to generate results for the transcoded frames at a second resolution lower than the first resolution;

[0325] upsampling the result in at least a vertical direction to a resolution higher than the second lower resolution to generate an upsampled result;

[0326] generating a decoded frame based at least on the upsampled result; and

[0327] The decoded frame is transferred to a frame buffer or a display device.

[0328] Clause 32. A computer-implemented method as described in clause 31, wherein the transmitting comprises transmitting the decoded frame to the frame buffer, and the computer-implemented method further comprises storing the corresponding decoded motion vector of the coded frame at the second lower resolution in the frame buffer.

[0329] Clause 33. The computer-implemented method of Clause 32, wherein the frame buffer includes the decoded frame at the higher resolution and does not include a lower resolution version of the decoded frame.

[0330] Clause 34. The computer-implemented method of Clause 31, wherein the upsampling comprises generating the upsampled results by a machine learning model.

[0331] Clause 35. The computer-implemented method of Clause 34, wherein generating the decoded frame comprises performing a residual operation based on the upsampled result at the higher resolution to generate the decoded frame.

[0332] Clause 36. The computer-implemented method of Clause 31 , wherein generating the decoded frame comprises performing a residual operation based on the upsampled result at the higher resolution to generate the decoded frame.

[0333] Clause 37. The computer-implemented method of Clause 31, wherein a viewer device comprises a decoder and the display device, and upsampling the result occurs within a post-processing operation of the decoder.

[0334] Clause 38. The computer-implemented method of Clause 37, wherein the post-processing operations include the upsampling and loop recovery operations performed at least in the vertical direction.

[0335] Clause 39. The computer-implemented method of Clause 38, wherein the post-processing operation further comprises performing a residual operation based on the upsampled result at the higher resolution to generate the decoded frame.

[0336] Clause 40. The computer-implemented method of Clause 31 , wherein upsampling the result in the vertical direction to the higher resolution occurs within a super-resolution mode of a decoder.

[0337] Clause 41. The computer-implemented method of Clause 31, wherein a header of the video includes super-resolution scaling parameters in the vertical direction.

[0338] Clause 42. A device comprising:

[0339] coupling to a display; and

[0340] A video decoder, the video decoder being configured to:

[0341] Receive the encoded frames of the video,

[0342] performing video transcoding on the transcoded frames of the video to generate results for the transcoded frames at a second resolution lower than the first resolution,

[0343] upsampling the result at least in a vertical direction to a resolution higher than the second lower resolution to generate an upsampled result,

[0344] generating a decoded frame based on the upsampled result, and

[0345] The decoded frame is transferred to a frame buffer or the display.

[0346] Clause 43. The apparatus of clause 42, wherein the video decoder transmits the decoded frame to the frame buffer and further stores corresponding decoded motion vectors for the coded frame at the second lower resolution in the frame buffer.

[0347] Clause 44. The apparatus of Clause 43, wherein the frame buffer includes the decoded frame at the higher resolution, and the video decoder does not include a lower resolution version of the decoded frame in the frame buffer.

[0348] Clause 45. The apparatus of clause 42, wherein the video decoder generates the upsampled result via a machine learning model.

[0349] Clause 46. The apparatus of Clause 42, wherein when the video decoder is in super-resolution mode, the video decoder upsamples the result to the higher resolution in at least the vertical direction.

[0350] Clause 47. The apparatus of Clause 42, wherein the video decoder further reads a header of the video, the header comprising the vertical super-resolution scaling parameters.

[0351] Figure 48 An example provider network (or "service provider system") environment is shown, according to some examples. Provider network 4800 can provide resource virtualization to customers via one or more virtualization services 4810, which allow customers to purchase, lease, or otherwise obtain instances 4812 of virtualized resources (including, but not limited to, computing resources and storage resources) implemented on devices within one or more provider networks in one or more data centers. A local Internet Protocol (IP) address 4816 can be associated with resource instance 4812; the local IP address is the internal network address of the resource instance 4812 on provider network 4800. In some examples, provider network 4800 can also provide public IP addresses 4814 and / or public IP address ranges (e.g., Internet Protocol version 4 (IPv4) or Internet Protocol version 6 (IPv6) addresses) that customers can obtain from provider 4800.

[0352] As is conventional, the provider network 4800, via the virtualization service 4810, can allow a customer of the service provider (e.g., a customer operating one or more client networks 4850A-4850C including one or more client devices 4852) to dynamically associate at least some public IP addresses 4814 assigned or allocated to the customer with a specific resource instance 4812 assigned to the customer. The provider network 4800 can also allow a customer to remap a public IP address 4814 previously mapped to one virtualized computing resource instance 4812 allocated to the customer to another virtualized computing resource instance 4812 also allocated to the customer. Using the virtualized computing resource instances 4812 and public IP addresses 4814 provided by the service provider, a customer of the service provider (e.g., an operator of the customer networks 4850A-4850C) can, for example, implement customer-specific applications and present the customer's applications on an intermediate network 4840 (e.g., the Internet). Other network entities 4820 on intermediate network 4840 can then generate traffic destined for destination public IP address 4814 published by customer networks 4850A-4850C; the traffic is routed to the service provider data center and there, via the network underlay, to local IP address 4816 of virtualized computing resource instance 4812, which is now mapped to destination public IP address 4814. Similarly, response traffic from virtualized computing resource instance 4812 can be routed back onto intermediate network 4840 to source entity 4820 via the network underlay.

[0353] As used herein, a local IP address refers to an internal or "private" network address of a resource instance, for example, in a provider network. A local IP address can be within an address block reserved by Request for Comments (RFC) 1918 of the Internet Engineering Task Force (IETF) and / or have an address format specified by IETF RFC4193, and can change within the provider network. Network traffic originating from outside the provider network is not routed directly to the local IP address; instead, the traffic uses a public IP address that is mapped to the local IP address of the resource instance. The provider network may include networking devices or appliances that provide network address translation (NAT) or similar functionality to perform mappings from public IP addresses to local IP addresses, and vice versa.

[0354] A public IP address is a variable network address on the Internet assigned to a resource instance by a service provider or customer. Traffic routed to a public IP address is translated, for example, via 1:1 NAT, and forwarded to the corresponding local IP address of the resource instance.

[0355] Some public IP addresses can be assigned to specific resource instances by the provider network infrastructure; these public IP addresses can be referred to as standard public IP addresses, or simply standard IP addresses. In some examples, the mapping of standard IP addresses to the local IP addresses of resource instances is the default launch configuration for all resource instance types.

[0356] At least some public IP addresses may be allocated to or obtained by customers of the provider network 4800; the customers may then assign their allocated public IP addresses to specific resource instances assigned to the customers. These public IP addresses may be referred to as customer public IP addresses, or simply customer IP addresses. Rather than being assigned to resource instances by the provider network 4800 as in the case of standard IP addresses, customer IP addresses may be assigned to resource instances by the customer, for example, via an API provided by the service provider. Unlike standard IP addresses, customer IP addresses are assigned to customer accounts and may be remapped to other resource instances by the respective customers as needed or desired. Customer IP addresses are associated with customer accounts rather than specific resource instances, and the customer controls the IP address until the customer chooses to release it. Unlike conventional static IP addresses, customer IP addresses allow customers to mask resource instance or availability zone failures by remapping the customer's public IP address to any resource instance associated with the customer account. For example, customer IP addresses enable customers to resolve issues with the customer's resource instances or software by remapping the customer IP address to an alternative resource instance.

[0357] Figure 49 4 is a block diagram of an example provider network that provides storage services and hardware virtualization services to customers according to some examples. Hardware virtualization service 4920 provides multiple computing resources 4924 (e.g., VMs) to customers. For example, computing resources 4924 can be rented or leased to customers of provider network 4900 (e.g., to customers implementing customer network 4950). Each computing resource 4924 can be configured with one or more local IP addresses. Provider network 4900 can be configured to route packets from the local IP addresses of computing resources 4924 to public internet destinations, and from public internet sources to the local IP addresses of computing resources 4924.

[0358] The provider network 4900 can provide a customer network 4950 coupled to the intermediary network 4940, for example, via a local network 4956, with the ability to implement a virtual computing system 4992 via a hardware virtualization service 4920 coupled to the intermediary network 4940 and the provider network 4900. In some examples, the hardware virtualization service 4920 can provide one or more APIs 4902 (e.g., web service interfaces) via which the customer network 4950 can access functionality provided by the hardware virtualization service 4920, for example, via a console 4994 (e.g., a web-based application, a standalone application, a mobile application, etc.). In some examples, at the provider network 4900, each virtual computing system 4992 at the customer network 4950 can correspond to a computing resource 4924 that is rented, leased, or otherwise provided to the customer network 4950.

[0359] From an instance of a virtual computing system 4992 and / or another customer device 4990 (e.g., via a console 4994), a customer may access functionality of a storage service 4910, e.g., via one or more APIs 4902, to access and store data (e.g., folders or "buckets," virtualized volumes, databases, etc.) from storage resources 4918A-4918N of a virtual data repository 4916 provided by a provider network 4900. In some examples, a virtualized data storage gateway (not shown) may be provided at the customer network 4950, which may cache at least some data (e.g., frequently accessed data or critical data) locally and may communicate with the storage service 4910 via one or more communication channels to upload new or modified data from the local cache, such that a master repository of data (the virtualized data repository 4916) is maintained. In some examples, a user via a virtual computing system 4992 and / or on another client device 4990 can mount and access virtualized data storage 4916 volumes via a storage service 4910 acting as a storage virtualization service, and these volumes can appear to the user as local (virtualized) storage 4998.

[0360] Although Figure 49 Although not shown, virtualized services may also be accessed from resource instances within provider network 4900 via API 4902. For example, a customer, device service provider, or other entity may access virtualized services via API 4902 from within a corresponding virtual network on provider network 4900 to request allocation of one or more resource instances within that virtual network or within another virtual network.

[0361] Descriptive System

[0362] In some examples, a system implementing part or all of the techniques for content indexing described herein may include a general-purpose computer system that includes or is configured to access one or more computer-accessible media, such as Figure 50 The computer system 5000 is shown. In the example shown, the computer system 5000 includes one or more processors 5010 coupled to a system memory 5020 via an input / output (I / O) interface 5030. The computer system 5000 also includes a network interface 5040 coupled to the I / O interface 5030. Although Figure 50 Computer system 5000 is shown as a single computing device, but in various examples, computer system 5000 may include one computing device or any number of computing devices configured to work together as a single computer system 5000 .

[0363] In various examples, computer system 5000 can be a uniprocessor system including one processor 5010 or a multiprocessor system including several processors 5010 (e.g., two, four, eight, or another suitable number). Processor 5010 can be any suitable processor capable of executing instructions. For example, in various examples, processor 5010 can be a general-purpose or embedded processor that implements any of a variety of instruction set architectures (ISAs), such as x86, ARM, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISAs. In a multiprocessor system, each of processors 5010 can typically, but not necessarily, implement the same ISA.

[0364] System memory 5020 can store instructions and data accessible by processor 5010. In various examples, system memory 5020 can be implemented using any suitable memory technology, such as random access memory (RAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash-type memory, or any other type of memory. In the illustrated example, program instructions and data that implement one or more desired functions (such as those methods, techniques, and data described above) are shown as stored within system memory 5020 as (e.g., multi-scale) ML code 5025 (e.g., executable to fully or partially implement an ML model or other operations discussed herein) and data 5026.

[0365] In one example, the I / O interface 5030 can be configured to coordinate I / O traffic between the processor 5010, the system memory 5020, and any peripheral devices in the device (including the network interface 5040 or other peripheral interfaces). In some examples, the I / O interface 5030 can perform any necessary protocol, timing, or other data transformations to convert data signals from one component (e.g., the system memory 5020) into a format suitable for use by another component (e.g., the processor 5010). In some examples, for example, the I / O interface 5030 can include support for devices attached via various types of peripheral buses (such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard). In some examples, for example, the functionality of the I / O interface 5030 can be split into two or more separate components, such as a northbridge and a southbridge. Furthermore, in some examples, some or all of the functionality of the I / O interface 5030 (such as the interface to the system memory 5020) can be incorporated directly into the processor 5010.

[0366] For example, the network interface 5040 may be configured to allow the computer system 5000 to communicate with other devices 5060 attached to one or more networks 5050 (e.g., such as Figure 1 The network interface 5040 may be configured to exchange data between the computer system and the other computer systems or devices shown. In various examples, for example, the network interface 5040 may support communication via any suitable wired or wireless general-purpose data network, such as various types of Ethernet networks. Additionally, the network interface 5040 may support communication via a telecommunications / telephone network, such as an analog voice network or a digital fiber-optic communication network, via a storage area network (SAN), such as a Fibre Channel SAN, or via any other suitable type of network and / or protocol.

[0367] In some examples, the computer system 5000 includes one or more offload cards 5070 (including one or more processors 5075 and possibly one or more network interfaces 5040) connected using an I / O interface 5030 (e.g., a bus implementing a version of the Peripheral Component Interconnect Express (PCI-E) standard or another interconnect such as Quick Path Interconnect (QPI) or Ultra Path Interconnect (UPI)). For example, in some examples, the computer system 5000 can act as a host electronic device hosting a compute instance (e.g., operating as part of a hardware virtualization service), and the one or more offload cards 5070 execute a virtualization manager that can manage the compute instance executing on the host electronic device. As an example, in some examples, the offload card 5070 can perform compute instance management operations such as pausing and / or unpausing a compute instance, starting and / or terminating a compute instance, performing memory transfer / copy operations, and the like. In some examples, these management operations can be performed by the offload card 5070 in cooperation with hypervisors (e.g., based on requests from the hypervisors) executed by the other processors 5010A-5010N of the computer system 5000. However, in some examples, the virtualization manager implemented by the offload card 5070 can adapt to requests from other entities (e.g., from the compute instances themselves) and may not cooperate with (or serve) any individual hypervisor.

[0368] In some examples, system memory 5020 can be an example of a computer-accessible medium configured to store program instructions and data as described above. However, in other examples, program instructions and / or data can be received, sent, or stored on different types of computer-accessible media. Generally speaking, computer-accessible media may include non-transitory storage media or storage media such as magnetic media or optical media, such as a disk or DVD / CD coupled to computer system 5000 via I / O interface 5030. Non-transitory computer-accessible storage media may also include any volatile or non-volatile media, such as RAM (e.g., SDRAM, double data rate (DDR) SDRAM, SRAM, etc.), read-only memory (ROM), etc., which may be included in some examples of computer system 5000 as system memory 5020 or another type of memory. In addition, computer-accessible media may include transmission media or signals, such as electrical signals, electromagnetic signals, or digital signals, transmitted via a communication medium (such as a network and / or wireless link), such as may be implemented via network interface 5040.

[0369] Figure 515100. The following is a logical arrangement of a set of general components of an example computing device 5100. In general, the computing device 5100 may also be referred to as an electronic device. The techniques shown in the figures and described herein may be implemented using code and data stored and executed on one or more electronic devices (e.g., client stations and / or server stations). Such electronic devices use computer-readable media to store and communicate (internally and / or with other electronic devices over a network) code and data, such as non-transitory computer-readable storage media (e.g., magnetic disks, optical disks, random access memory (RAM), read-only memory (ROM), flash memory devices, phase-change memory) and transient computer-readable communication media (e.g., electrical, optical, acoustic, or other forms of propagated signals, such as carrier waves, infrared signals, digital signals). In addition, such electronic devices include hardware such as: a set of one or more processors 5102 (e.g., where the processor is a microprocessor, a controller, a microcontroller, a central processing unit, a digital signal processor, an application specific integrated circuit, a field programmable gate array, other electronic circuits, or a combination of one or more of the foregoing), the one or more processors coupled to one or more other components, such as one or more non-transitory machine-readable storage media (e.g., memory 5104) to store code (e.g., instructions 5110, e.g., that implement the content distribution service disclosed herein); and a set of one or more wired or wireless network interfaces 5108 that allow the electronic device to transmit data to and receive data from other computing devices, typically across one or more networks (e.g., a local area network (LAN), the Internet). The coupling of the set of processors and other components is typically performed through one or more interconnects (e.g., buses and possibly bridges) within the electronic device. Thus, the non-transitory machine-readable storage medium (e.g., memory 5104) of a given electronic device typically stores code (e.g., instructions 5110) for execution on the set of one or more processors 5102 of that electronic device. One or more portions of the various examples may be implemented using different combinations of software, firmware, and / or hardware.

[0370] The computing device 5100 may include some type of display element 5106, such as a touch screen or liquid crystal display (LCD), but many devices, such as portable media players, may communicate information via other means (such as through audio speakers), and other types of devices, such as server-side stations, may not have a display element 5106 at all. As discussed, some computing devices used in some examples include at least one input and / or output component 5112 capable of receiving input from a user. This input component may include, for example, a button, a touchpad, a touch screen, a scroll wheel, a joystick, a keyboard, a mouse, a keypad, or any other such device or element by which a user can enter commands to the device. However, in some examples, such devices may be controlled by a combination of visual and / or audio commands and utilize microphones, cameras, sensors, etc., so that a user can control the device without having to physically contact the device.

[0371] As discussed, different approaches can be implemented in various environments according to the described examples. For example, Figure 52 An example of an environment 5200 for implementing aspects according to various examples is shown. For example, in some examples, messages are Hypertext Transfer Protocol (HTTP) requests received by a web server (e.g., web server 5206), and a user can interact with a provider network via an electronic device via a web portal provided by web server 5206 and application server 5208. As will be appreciated, although a web-based environment is used for illustrative purposes, different environments can be used to implement the various examples as appropriate. The system includes an electronic client device 5202 (also referred to as a client device), which can be any suitable device operable to send and receive requests, messages, or information over an appropriate network 5204 and communicate the information back to the user of the device 5202. Examples of such client devices include personal computers (PCs), mobile phones, handheld messaging devices, laptops, set-top boxes, personal data assistants, e-book readers, wearable electronic devices (e.g., glasses, wristbands, monitors), etc. The one or more networks 5204 can include any suitable network, including an intranet, the Internet, a cellular network, a local area network, or any other such network or combination thereof. The components used by such a system may depend, at least in part, on the type of network and / or environment selected. The protocols and components used to communicate via such a network are well known and will not be discussed in detail herein. Communication on the network may be achieved via wired or wireless connections and combinations thereof. In this example, the network 5204 comprises the Internet because the environment includes a web server 5206 for receiving requests and providing content in response to the requests, but alternative devices serving similar purposes may be used for other networks, as will be apparent to those skilled in the art.

[0372] The illustrative environment includes at least one application server 5208 and a data repository 5210. It should be understood that there may be several application servers, layers, or other elements, processes, or components that may be linked or otherwise configured that can interact to perform tasks such as obtaining data from an appropriate data repository. As used herein, the term "data repository" refers to any device or combination of devices capable of storing, accessing, and retrieving data, which may include any combination and any number of data servers, databases, data storage devices, and data storage media in any standard, distributed, or clustered environment. The application server 5208 may include any appropriate hardware and software for integrating with the data repository 5210 as needed to execute various aspects of one or more applications of the client device 5202 and for handling most of the data access and business logic of the applications. The application server 5208 cooperates with the data repository 5210 to provide access control services and is capable of generating content such as text, graphics, audio, video, etc. to be transmitted to the client device 5202. In this example, the content can be provided to the user by the web server in the form of Hypertext Markup Language (HTML), Extensible Markup Language (XML), JavaScript Object Notation (JSON), or another suitable unstructured or structured language. The handling of all requests and responses and the transfer of content between the client device 5202 and the application server 5208 can be handled by the web server 5206. It should be understood that the web server 5206 and the application server 5208 are not required but are merely exemplary components, as the structured code discussed herein can be executed on any suitable device or host as discussed elsewhere herein.

[0373] The data repository 5210 may include several separate data tables, databases, or other data storage mechanisms and media for storing data related to a particular aspect. For example, the data repository shown includes mechanisms for storing production data 5212 and user information 5216, which can be used to provide content for the production side. The data repository 5210 is also shown as including mechanisms for storing log or session data 5214. It should be understood that there may be many other aspects that may need to be stored in the data repository, such as page image information and access rights information, which may be stored in any of the mechanisms listed above or in additional mechanisms within the data repository 5210, as appropriate. The data repository 5210 can operate through its associated logic to receive instructions from the application server 5208 and, in response to these instructions, retrieve, update, or otherwise process data. In one example, a user may submit a search request for a certain type of project. In this case, the data repository 5210 may access the user information 5216 to verify the user's identity and may access the production data 5212 to obtain information about projects of that type. The information may then be returned to the user, such as in a results list, on a web page that the user can view via a browser on the user device 5202. Information about a particular item of interest may be viewed in a dedicated page or window in the browser.

[0374] The web server 5206, application server 5208, and / or data repository 5210 may be implemented by one or more electronic devices 5220, which may also be referred to as electronic server devices or server end stations, and may or may not be located in different geographical locations. Each of the one or more electronic devices 5220 may include an operating system that provides executable program instructions for the general management and operation of the device, and will typically include a computer-readable medium storing instructions that, when executed by the device's processor, allow the device to perform its intended functions. Suitable implementations for the operating system and general functionality of the device are known or commercially available, and can be readily implemented by one of ordinary skill in the art, particularly in light of the disclosure herein.

[0375] In one example, the environment is a distributed computing environment utilizing several computer systems and components that are interconnected via communication links using one or more computer networks or direct connections. However, it will be understood by those skilled in the art that such a system may be implemented in a variety of environments with more than Figure 52 The system shown may operate equally well with fewer or more components. Figure 52 The depiction of environment 5200 in should be regarded as illustrative in nature and not limiting the scope of the present disclosure.

[0376] The various examples discussed or presented herein can be implemented in a wide variety of operating environments, which in some cases may include one or more user computers, computing devices, or processing devices that can be used to operate any of a number of applications. User or client devices may include any of a number of general-purpose personal computers, such as desktop or laptop computers running standard operating systems, and cellular, wireless, and handheld devices running mobile software and capable of supporting a number of networking and messaging protocols. Such systems may also include a number of workstations running any of a variety of commercially available operating systems and other known applications for purposes such as development and database management. These devices may also include other electronic devices, such as virtual terminals, thin clients, gaming systems, and / or other devices capable of communicating via a network.

[0377] Most examples utilize at least one network familiar to those skilled in the art to support communications using any of a variety of commercially available protocols, such as Transmission Control Protocol / Internet Protocol (TCP / IP), File Transfer Protocol (FTP), Universal Plug and Play (UPnP), Network File System (NFS), Common Internet File System (CIFS), Extensible Messaging and Presence Protocol (XMPP), AppleTalk, etc. The network may include, for example, a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), the Internet, an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network, and any combination thereof.

[0378] In the example of utilizing a web server, the web server can run any of a variety of server or middle-tier applications, including HTTP servers, file transfer protocol (FTP) servers, common gateway interface (CGI) servers, data servers, Java servers, business application servers, etc. The server can also execute programs or scripts in response to requests from user devices, such as by executing programs that can be implemented in any programming language (such as The one or more servers may include one or more web applications written in one or more scripts or programs in C, C#, or C++, or any scripting language (such as Perl, Python, PHP, or TCL), or any combination thereof. The one or more servers may also include a database server, including but not limited to database servers available from Oracle(R), Microsoft(R), Sybase(R), IBM(R), etc. The database server may be relational or non-relational (e.g., "NoSQL"), distributed or non-distributed, etc.

[0379] The environment may include a variety of data repositories as discussed above, as well as other memories and storage media. These may reside in various locations, such as on (and / or in) storage media local to one or more computers, or on storage media remote from any or all computers on the network. In a specific set of examples, the information may reside in a storage area network (SAN) familiar to those skilled in the art. Similarly, any necessary files for performing the functions attributed to a computer, server, or other network device may be stored locally and / or remotely, as appropriate. In the case where the system includes computerized devices, each such device may include hardware elements that can be electrically coupled via a bus, including, for example, at least one central processing unit ("CPU"), at least one input device (e.g., a mouse, keyboard, controller, touch screen, or keypad), and / or at least one output device (e.g., a display device, printer, or speaker). Such a system may also include one or more storage devices, such as hard drives, optical storage devices, and solid-state storage devices such as random access memory (RAM) or read-only memory (ROM), as well as removable media devices, memory cards, flash memory cards, and the like.

[0380] Such devices may also include a computer-readable storage medium reader, a communication device (e.g., a modem, a network card (wireless or wired), an infrared communication device, etc.) and a working memory as described above. The computer-readable storage medium reader may be connected to or configured to receive a computer-readable storage medium, which represents a remote, local, fixed and / or removable storage device and storage medium for temporarily and / or longer-term accommodating, storing, transmitting and retrieving computer-readable information. The system and various devices will typically also include a number of software applications, services or other elements located within at least one working memory device, including an operating system and application programs, such as a client application or a web browser. It should be understood that alternative examples may have many variations different from those described above. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software (including portable software, such as applets), or both. In addition, connections to other computing devices such as network input / output devices may be employed.

[0381] The storage media and computer-readable media used to hold the code or code portions may include any suitable media known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile media, removable and non-removable media implemented in any method or technology to store and / or transmit information (such as computer-readable instructions, data structures, program code or other data), including RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk-read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by the system device. Based on this disclosure and the teachings provided herein, those skilled in the art will understand other ways and / or methods to implement the various examples.

[0382] In the foregoing description, various examples have been described. For illustrative purposes, specific configurations and details have been set forth to provide a thorough understanding of the examples. However, it will be apparent to those skilled in the art that the examples can be practiced without these specific details. Furthermore, well-known features may be omitted or simplified to avoid obscuring the described examples.

[0383] This document may use bracketed text and boxes with dashed borders (e.g., large dashes, small dashes, dot dashes, and dots) to illustrate optional operations that add additional features to some examples. However, this notation should not be taken to mean that these are the only options or optional operations, and / or that in some examples, boxes with solid borders are not optional.

[0384] Reference numerals with suffix letters (e.g., 4918A-4918N) may be used to indicate that there may be one or more instances of the referenced entity in various examples, and when multiple instances are present, each instance need not be identical but may share some general characteristics or function in a common manner. Furthermore, unless expressly indicated to the contrary, the use of a particular suffix is not intended to imply the presence of a particular quantity of the entity. Thus, in various examples, two entities using the same or different suffix letters may or may not have the same number of instances.

[0385] References to "one example," "an example," "some examples," etc. indicate that the described examples may include a particular feature, structure, or characteristic, but not every example may include the particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same example. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an example, it should be understood that it is within the knowledge of those skilled in the art to implement such feature, structure, or characteristic in conjunction with other examples, whether or not explicitly described.

[0386] Furthermore, in the various examples described above, unless specifically stated otherwise, disjunctive language such as the phrase "at least one of A, B, or C" is intended to be understood to mean A, B, or C, or any combination thereof (e.g., A, B, and / or C). Thus, disjunctive language is neither intended nor should be understood to imply that a given example requires that at least one of A, at least one of B, or at least one of C each be present.

[0387] The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the disclosure as set forth in the claims.

Claims

1. A computer-implemented method comprising: receiving the video at a content distribution service; performing encoding, by the content distribution service, on frames of the video, the encoding converting the frames from a pixel domain to a transform domain and back to the pixel domain to generate first pixel values and first residuals for blocks of the frames at a first resolution; generating, by a machine learning model of the content distribution service, a first feature set based on inputs of the first pixel value and the first residual of the block at the first resolution; generating, by the machine learning model of the content distribution service, a second feature set for input of a second pixel value and a second residual of the block at a second lower resolution; upsampling the second feature set to the first resolution to generate an upsampled second feature set; generating a modified version of the frame based on the first feature set and the upsampled second feature set; as well as The modified version of the frame is transmitted to a frame buffer or from the content distribution service to a viewer device.

2. The computer-implemented method of claim 1 , further comprising: generating, by the machine learning model of the content distribution service, a third feature set for input of third pixel values and third residuals of the block at a third resolution lower than the second lower resolution; as well as upsampling the third feature set to the first resolution to generate an upsampled third feature set, Wherein generating the modified version of the frame is based on the first feature set, the upsampled second feature set, and the upsampled third feature set.

3. The computer-implemented method of claim 1, wherein generating the first feature set, generating the second feature set, and generating the modified version of the frame occur within a loop filter of an encoder.

4. A computer-implemented method comprising: performing video coding processing on a frame of a video, the video coding processing generating first pixel values and first residuals for a block of the frame; Generating a first feature set by a machine learning model based on the first pixel value and the first residual of the block at a first resolution; generating, by the machine learning model, a second feature set for input of a second pixel value and a second residual of the block at a second lower resolution; upsampling the second feature set to the first resolution to generate an upsampled second feature set; generating a modified version of the frame based on the first feature set and the upsampled second feature set; as well as The modified version of the frame is transmitted to a frame buffer or a display device.

5. The computer-implemented method of claim 4 , further comprising: generating, by the machine learning model, a third feature set for input of a third pixel value and a third residual of the block at a third resolution lower than the second lower resolution; as well as upsampling the third feature set to the first resolution to generate an upsampled third feature set, Wherein generating the modified version of the frame is based on the first feature set, the upsampled second feature set, and the upsampled third feature set.

6. The computer-implemented method of claim 4, wherein generating the first feature set, generating the second feature set, and generating the modified version of the frame occur within a loop filter of an encoder.

7. The computer-implemented method of claim 4, wherein a viewer device comprises a decoder and the display device, and generating the first feature set, generating the second feature set, and generating the modified version of the frame occur within a loop filter of the decoder.

8. The computer-implemented method of claim 7, further comprising: determining an indication of a subset of blocks of a frame to be processed by a machine learning model of the decoder; as well as The indication is sent to the decoder to cause the decoder to process the subset of blocks of the frame through the machine learning model of the decoder.

9. The computer-implemented method of claim 4, wherein generating the first feature set, generating the second feature set, and generating the modified version of the frame occur in a post-processor separate from any encoder and any decoder.

10. The computer-implemented method of claim 4, further comprising: Prior to generating the second set of features, the block is downsampled from the first resolution to the second, lower resolution.

11. The computer-implemented method of claim 10, wherein the downsampling comprises performing a strided convolution on the block at the first resolution.

12. The computer-implemented method of claim 4, wherein the upsampling comprises interleaving a plurality of channels into one channel.

13. The computer-implemented method of claim 4, wherein generating the modified version of the frame comprises performing a cross-component sample offset operation.

14. The computer-implemented method of claim 4, further comprising: One of the modified version of the block and another version of the block is selected as input to a cross-component sample offset operation.

15. A non-transitory computer-readable medium storing code that, when executed by an apparatus, causes the apparatus to perform a method comprising: performing video coding processing on a frame of a video, the video coding processing generating first pixel values and first residuals for a block of the frame; Generating a first feature set by a machine learning model based on the first pixel value and the first residual of the block at a first resolution; generating, by the machine learning model, a second feature set for input of a second pixel value and a second residual of the block at a second lower resolution; upsampling the second feature set to the first resolution to generate an upsampled second feature set; generating a modified version of the frame based on the first feature set and the upsampled second feature set; as well as The modified version of the frame is transmitted to a frame buffer or a display device.

Images