METHOD AND DEVICE FOR ENCODING VIDEO DATA, AND COMPUTER-READABLE STORAGE MEDIUM

Abstract

A device for encoding video data can be configured to encode a set of input frames of the video data with an image analysis neural network to generate a corresponding set of output frames, wherein the set of output frames includes a first output frame and additional output frames, wherein the first output frame temporally precedes the additional output frames; determine a modification to the first output frame that results in an optimized amount of distortion between the set of input frames and decoded versions of the corresponding set of output frames; update the first output frame based on the determined modification; and output a bitstream that includes the updated first output frame.

Description

1 / 46 “METHOD AND DEVICE FOR ENCODING VIDEO DATA, AND COMPUTER-READABLE STORAGE MEDIUM

[0001] This application claims priority over U.S. Patent Application No. 18 / 188,268, filed March 22, 2023, the entire contents of which are incorporated by reference. TECHNICAL FIELD

[0002] This disclosure relates to media encoding and decoding, including the encoding and decoding of images and video data. BACKGROUND

[0003] Digital media capabilities can be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radiotelephones, so-called smartphones, video teleconferencing devices, video streaming devices, and the like. Digital video devices implement video coding techniques, such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265 / High Efficiency Video Coding (HEVC), ITU-T H.266 / Versatile Video Coding (VVC) and extensions of these standards, as well. Petition 870250081169, dated 10 / 09 / 2025, page 8 / 118 2 / 46 proprietary video codecs / formats, such as AOMedia Video 1 (AV1), which was developed by the Alliance for Open Media. Video devices can transmit, receive, encode, decode, and / or store digital video information more efficiently by implementing these video encoding techniques. SUMMARY

[0004] An E2E-NVC can increase the quality of the decoded video by inserting I-frames more frequently, but this typically results in a rate-distortion tradeoff that is inferior to that of HEVC and VVC encoders. Therefore, in order to reduce the bitrate, E2E-NVCs will train on a smaller number of frames, such as 3 to 7, hoping that this limited training will produce desirable results over a much larger sequence of frames. Typically, however, training on a substantially smaller number of frames than what is included in the P-frame sequence results in increasing distortion as the media decoder progresses through the P-frame sequence. This variable level of distortion can produce a user experience that, in many cases, is even worse than consistently high distortion.

[0005] This disclosure describes sliding window rate-distortion optimization techniques that address the problems introduced above. As will be described in more detail below, this disclosure describes techniques for encoding a set of input frames with an image analysis neural network to generate a corresponding set of output frames. By using a window Petition 870250081169, dated 10 / 09 / 2025, p. 9 / 118 3 / 46 slider to determine a modification in a first output frame that results in an optimized amount of distortion between the input frame set and the decoded versions of the corresponding output frame set, the techniques of this disclosure can enable E2ENVCs to use P-frame sequences that are comparable in length to the sequences used by conventional video encoders without increased distortion as the E2E-NVC advances through the P-frame sequence.

[0006] According to one example, a video data encoding method includes encoding a set of input frames of video data with an image analysis neural network to generate a corresponding set of output frames, wherein the output frame set includes a first output frame and additional output frames, wherein the first output frame temporally precedes the additional output frames; determining a modification to the first output frame that results in an optimized amount of distortion between the input frame set and the decoded versions of the corresponding output frame set; updating the first output frame based on the determined modification; and emitting a bitstream that includes the updated first output frame.

[0007] According to another example, a device for encoding video data includes a memory configured to store video data; one or more processors implemented in a circuit array and configured to: encode a set of input frames of video data with an image analysis neural network to generate a corresponding set of frames of Petition 870250081169, dated 10 / 09 / 2025, page 10 / 118 4 / 46 output, wherein the set of output frames includes a first output frame and additional output frames, wherein the first output frame temporally precedes the additional output frames; determine a modification to the first output frame that results in an optimized amount of distortion between the input frame set and the decoded versions of the corresponding set of output frames; update the first output frame based on the determined modification; and emit a bitstream that includes the updated first output frame.

[0008] According to another example, a computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to encode a set of input frames of video data with an image analysis neural network to generate a corresponding set of output frames, wherein the set of output frames includes a first output frame and additional output frames, wherein the first output frame temporally precedes the additional output frames; determine a modification in the first output frame that results in an optimized amount of distortion between the input frame set and the decoded versions of the corresponding set of output frames; update the first output frame based on the determined modification; and output a bitstream that includes the updated first output frame.

[0009] According to another example, a device for encoding video data includes means for encoding a set of input frames of the video data. Petition 870250081169, dated 10 / 09 / 2025, page 11 / 118 5 / 46 using an image analysis neural network to generate a matching set of output frames, wherein the set of output frames includes a first output frame and additional output frames, wherein the first output frame temporally precedes the additional output frames; means to determine a modification in the first output frame that results in an optimized amount of distortion between the input frame set and the decoded versions of the matching set of output frames; means to update the first output frame based on the determined modification; and means to output a bitstream that includes the updated first output frame.

[0010] The details of one or more examples are set forth in the attached drawings and the description below. Other attributes, objectives and advantages will become apparent from the description, drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram illustrating an example media encoding and decoding system that can perform the techniques of this disclosure.

[0012] Figure 2 illustrates the differences between conventional media coding and neural-based media coding.

[0013] Figure 3 shows an example of a set of input frames encoded with an image analysis neural network to generate a set of output frames.

[0014] Figure 4 is a block diagram illustrating an example neural-based image codec.

[0015] Figure 5 is a flowchart that illustrates Petition 870250081169, dated 10 / 09 / 2025, page 12 / 118 6 / 46 an example process for encoding a set of input frames according to the techniques of this disclosure. DETAILED DESCRIPTION

[0016] This disclosure describes techniques for encoding, for example, encoding and / or decoding, media data (e.g., images or videos) using neural network-based media encoding techniques. Both conventional video encoding and neural network-based video encoding utilize a combination of intracoding and intercoding. When performing intracoding, a media encoder encodes a portion of video data, such as a frame or portion thereof, using only data from the same frame as the portion being encoded. When performing intercoding, a media encoder can encode the portion of video data using data from previously encoded frames, in addition to using data from the same frame as the portion being encoded.Thus, an intracoded video data frame, also called an I-frame, generally refers to a frame where all the data necessary to decode the frame is included within the frame, while an intercoded frame, also called a predicted frame or P-frame, requires data from previously encoded frames. Although the terms frame and image may have specific meanings in certain codecs or standards, this disclosure uses the terms interchangeably in a generic way to refer to any type of intermediate or final output image.

[0017] In both conventional video coding and neural network-based video coding, P-frames provide coding efficiency. Petition 870250081169, dated 10 / 09 / 2025, page 13 / 118 7 / 46 significantly better compared to I-frames. A certain number of I-frames may be necessary, however, to achieve a desired level of encoding quality, as well as to support random access and other similar features. A conventional video encoder, according to the HEVC or VVC standard, for example, may include an I-frame with a frequency of up to once every 30 frames, but in many encoding scenarios, it may include an I-frame with a much lower frequency, such as once every 120 frames, or even lower, once every 600 frames. End-to-end neural video encoders (E2E-NVCs), however, typically do not perform well with such large P-frame sequences.One reason why E2E-NVCs do not perform well on such large P-frame sequences is that the training time on large P-frame sequences is incompatible with low-latency encoding applications, and even then it is generally not practical for non-low-latency encoding applications due to time, memory, and processing requirements.

[0018] An E2E-NVC can increase the quality of the decoded video by inserting I-frames more frequently, but this typically results in a rate-distortion tradeoff that is inferior to that of HEVC and VVC encoders. Therefore, in order to reduce the bitrate, E2E-NVCs will train on a smaller number of frames, such as 3 to 7, hoping that this limited training will produce desirable results on a much larger frame sequence. Typically, however, training on a substantially smaller number of frames than what is included in the P-frame sequence results in increasing distortion. Petition 870250081169, dated 10 / 09 / 2025, p. 14 / 118 8 / 46 as the media decoder advances through the P-frame sequence. This variable level of distortion can produce a user experience that, in many cases, is even worse than consistently high distortion.

[0019] This disclosure describes sliding window rate-distortion optimization techniques that address the problems introduced above. As will be described in more detail below, this disclosure describes techniques for encoding an input frameset with an image analysis neural network to generate a corresponding output frameset. By using a sliding window to determine a modification in a first output frame that results in an optimized amount of distortion between the input frameset and the decoded versions of the corresponding output frameset, the techniques in this disclosure can enable E2ENVCs to use P-frame sequences that are comparable in length to the sequences used by conventional video encoders without increased distortion as the E2E-NVC progresses through the P-frame sequence.

[0020] Figure 1 is a block diagram illustrating an example of a media encoding and decoding system 100 that can perform the techniques of this disclosure. In the context of this disclosure, media can include any digital file to be compressed, including video and / or image data. The example techniques of this disclosure generally refer to the process of encoding (encoding and / or decoding) video and / or image data. Although the examples in Figure 1 are described with reference to encoding and decoding Petition 870250081169, dated 10 / 09 / 2025, page 15 / 118 9 / 46 of media, the techniques of this application are equally applicable to the encoding and decoding of any type of data file using neural-based compression techniques.

[0021] As shown in Figure 1, the system 100 includes a source device 102 that provides encoded media data to be decoded and displayed by a destination device 116 in this example. In particular, the source device 102 provides the media data to the destination device 116 via a computer-readable medium 110. The source device 102 and the destination device 116 may comprise any of a wide range of devices, including desktop computers, notebook computers (i.e., laptops), mobile devices, tablet computers, set-top boxes, telephone handsets such as smartphones, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, broadcast receiving devices, or similar devices.In some cases, the source device 102 and the destination device 116 may be equipped for wireless communication and, in this way, may be referred to as wireless communication devices.

[0022] In the example in Figure 1, the source device 102 includes a media source 104, memory 106, a media encoder 200, and an output interface 108. The destination device 116 includes an input interface 122, a media decoder 400, a memory 120, and a display device 118. According to this disclosure, the media encoder 200 of the source device 102 can be configured to apply the techniques for optimization of Petition 870250081169, dated 10 / 09 / 2025, page 16 / 118 10 / 46 sliding window distortion rate described in the present invention. The source device 102 represents an example of a media encoding device, while the destination device 116 represents an example of a media decoding device. In other examples, a source device and a destination device may include other components or arrangements. For example, the source device 102 may receive media data from an external media source, such as an external camera. Similarly, the destination device 116 may interface with an external display device, rather than including an integrated display device.

[0023] As shown in Figure 1, system 100 is merely an example. In general, any digital media encoding and / or decoding device can perform entropic coding techniques of a neural-based media compression system. Source device 102 and destination device 116 are merely examples of such encoding devices in which source device 102 generates encoded media data for transmission to destination device 116. This disclosure refers to an encoding device as a device that performs the encoding (encoding and / or decoding) process of data. Thus, media encoder 200 and media decoder 400 represent examples of encoding devices, in particular, a media encoder and a media decoder, respectively.In some examples, the source device 102 and the destination device 116 may operate in a substantially symmetrical manner, such that each... Petition 870250081169, dated 10 / 09 / 2025, page 17 / 118 11 / 46 between the source device 102 and the destination device 116 includes media encoding and decoding components. In this way, the system 100 can support unidirectional or bidirectional media transmission between the source device 102 and the destination device 116, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0024] In general, the media source 104 represents a media data source (i.e., raw, unencoded media data) and provides a sequential series of images (also called frames) of the media data to the media encoder 200, which encodes data into images. The media source 104 of the source device 102 may include a video capture device, such as a video camera, a video file containing previously captured raw video, and / or a video feed interface for receiving video from a video content provider. As an additional alternative, the media source 104 may generate computer graphics-based data, such as the source media, or a combination of live media, archived media, and computer-generated media. In each case, the media encoder 200 encodes the captured, pre-captured, or computer-generated media data.The media encoder 200 can rearrange the images from the received order (sometimes called the display order) into an encoding order for encoding. The media encoder 200 can generate a bitstream that includes encoded media data. The source device 102 can then output the encoded media data via the output interface 108 to the computer-readable medium 110 for reception and / or a. Petition 870250081169, dated 10 / 09 / 2025, page 18 / 118 12 / 46 recovery, for example, via input interface 122 of destination device 116.

[0025] Memory 106 of source device 102 and memory 120 of destination device 116 represent general-purpose memories. In some examples, memories 106 and 120 may store raw media data, for example, raw media from media source 104 and raw and decoded media data from media decoder 400. Additionally or alternatively, memories 106 and 120 may store executable software instructions, for example, by media encoder 200 and media decoder 400, respectively. Although memory 106 and memory 120 are shown separately from media encoder 200 and media decoder 400 in this example, it should be understood that media encoder 200 and media decoder 400 may also include internal memories for functionally similar or equivalent purposes.In addition, memories 106 and 120 can store encoded media data, for example, output from media encoder 200 and input to media decoder 400. In some examples, portions of memories 106 and 120 can be allocated as one or more buffers, for example, to store raw, decoded, and / or encoded media data.

[0026] The computer-readable medium 110 may represent any type of medium or device with the ability to carry encoded media data from the source device 102 to the destination device 116. In one example, the computer-readable medium 110 represents a communication medium to enable the source device 102 to transmit encoded media data directly to Petition 870250081169, dated 10 / 09 / 2025, page 19 / 118 13 / 46 the destination device 116 in real time, for example, via a radio frequency network or computer-based network. The output interface 108 can modulate a transmission signal that includes the encoded media data, and the input interface 122 can demodulate the received transmission signal according to a communication standard, such as a wireless communication protocol. The communication medium can comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium can be part of a packet-based network, such as a local area network, a wide area network, or a global network, such as the Internet. The communication medium can include routers, switches, base stations, or any other equipment that may be useful in facilitating communication between the source device 102 and the destination device 116.

[0027] In some examples, source device 102 may output encoded data from output interface 108 to storage device 112. Similarly, destination device 116 may access encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as a hard disk, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other digital storage medium suitable for storing encoded media data.

[0028] In some examples, the source device 102 may output encoded media data to the Petition 870250081169, dated 10 / 09 / 2025, page 20 / 118 14 / 46 file server 114 or other intermediate storage device that can store the encoded media data generated by the source device 102. The destination device 116 can access stored media data from the file server 114 via streaming or download.

[0029] The file server 114 can be any type of server device with the ability to store encoded media data and transmit that encoded media data to the destination device 116.File server 114 can represent a web server (for example, for a website), a server configured to provide a file transfer protocol service (such as File Transfer Protocol (FTP) or File Delivery over Unidirectional Transport (FLUTE)), a content delivery network (CDN) device, a hypertext transfer protocol (HTTP) server, a multimedia broadcast multicast service (MBMS) or enhanced MBMS (eMBMS) server, and / or a network attached storage (NAS) device.File server 114 may, additionally or alternatively, implement one or more HTTP streaming protocols, such as dynamic adaptive streaming over HTTP (DASH), HTTP live streaming (HLS), real-time streaming protocol (RTSP), streaming. Petition 870250081169, dated 10 / 09 / 2025, page 21 / 118 15 / 46 dynamic HTTP or similar.

[0030] The target device 116 can access encoded media data from the file server 114 through any standard data connection, including an internet connection. This can include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., digital subscriber line (DSL), cable modem, etc.), or a combination of both that is suitable for accessing encoded media data stored on the file server 114. The input interface 122 can be configured to operate according to any one or more of the various protocols discussed above to retrieve or receive media data from the file server 114, or other protocols to retrieve media data.

[0031] Output interface 108 and input interface 122 may represent wireless transmitters / receivers, modems, wired network communication components (e.g., Ethernet cards), wireless communication components operating in accordance with any of a variety of Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards, or other physical components. In examples where output interface 108 and input interface 122 comprise wireless components, output interface 108 and input interface 122 may be configured to transfer data, such as encoded media data, in accordance with a cellular communication standard, such as fourth generation (4G), fourth generation long-term evolution (4G-LTE), advanced LTE, fifth generation (5G). Petition 870250081169, dated 10 / 09 / 2025, page 22 / 118 16 / 46 fifth generation) or similar. In some examples where the output interface 108 comprises a wireless transmitter, the output interface 108 and the input interface 122 may be configured to transfer data, such as encoded media data, according to other wireless standards, such as an IEEE 802.11 specification, an IEEE 802.15 specification (e.g., ZigBee™), a Bluetooth™ standard, or similar. In some examples, the source device 102 and / or the destination device 116 may include their respective system-on-a-chip (SoC) devices. For example, source device 102 may include a SoC device to perform the functionality assigned to media encoder 200 and / or output interface 108, and destination device 116 may include a SoC device to perform the functionality assigned to media decoder 400 and / or input interface 122.

[0032] The techniques of this disclosure can be applied to media encoding in support of any of a variety of multimedia applications, such as over-the-air television broadcasts, cable television broadcasts, satellite television broadcasts, Internet streaming media such as dynamic adaptive streaming over HTTP (DASH - dynamic adaptive streaming hypertext transfer protocol), digital media that is encoded on a data storage medium, decoding of digital media stored on a data storage medium, or other applications. The techniques of this disclosure may be of particular benefit to low-latency applications, but are not limited to low-latency applications.

[0033] Input interface 122 of Petition 870250081169, dated 10 / 09 / 2025, page 23 / 118 17 / 46 Destination device 116 receives an encoded media bitstream from the computer-readable medium 110 (e.g., a communication medium, storage device 112, file server 114, or similar). The encoded media bitstream may include signaling information defined by the media encoder 200, which is also used by the media decoder 400. The display device 118 displays decoded images of the decoded media data to a user. The display device 118 may represent any of several display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0034] Although not shown in Figure 1, in some examples, the media encoder 200 and the media decoder 400 can each be integrated with an audio encoder and / or audio decoder and may include suitable MUX-DEMUX units, or other hardware and / or software, to handle multiplexed streams that include audio and media in a common data stream.

[0035] The media encoder 200 and the media decoder 400 can each be implemented as any one of a variety of suitable encoder and / or decoder circuit sets, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs). Petition 870250081169, dated 10 / 09 / 2025, page 24 / 118 18 / 46 arrays), discrete logic, software, hardware, firmware, or any combination thereof. When the techniques are implemented partially in software, a device may store instructions for the software on a suitable non-transient, computer-readable medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Each of the 200 media encoder and 400 media decoder may be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder / decoder (CODEC) in a respective device. A device that includes a 200 media encoder and / or a 400 media decoder may comprise an integrated circuit, a microprocessor, and / or a wireless communication device, such as a mobile phone.

[0036] Image and video (e.g., media) compression methods based on neural networks can be competitive with current standards and provide several additional advantages; however, as introduced above, current neural network-based media encoders do not handle longer P-frame sequences well, which impairs overall performance.

[0037] As shown in Figure 2, the design phase of conventional and neural-based coding techniques includes the data organization process 201 and the statistical data modeling process 202. For conventional media coding techniques, such as H.265, the data organization process 201 includes the definition of coding elements (e.g., syntax elements and modes of Petition 870250081169, dated 10 / 09 / 2025, page 25 / 118 19 / 46 encoding). For neural-based encoding processes, the data organization process 201 includes the training and development of image / video autoencoder networks (e.g., media). For the statistical data modeling process 202, conventional encoding techniques include context-defining encoding, for example, for an entropy-based encoding process, such as context-adaptive binary arithmetic encoding. For neural-based encoding processes, the statistical data modeling process 202 includes the development and training of a hyperpriority neural network.

[0038] Both conventional and neural-based coding techniques include an implementation phase that includes the 204 probability determination process and the 206 bit-process conversion of information. Both the 204 probability determination process and the 206 bit-process conversion of information are based on the data to be encoded. For conventional coding, the 204 probability determination process (e.g., a bit-rate assignment process) includes adaptive context probability estimation. For neural-based coding, the 204 probability determination process includes the determination of probability distribution parameters, as will be described in detail below. Both conventional and neural-based coding techniques use an arithmetic and / or entropy encoder (e.g., either an arithmetic or entropy encoder or decoder) for the conversion of information into a bit-process. Petition 870250081169, dated 10 / 09 / 2025, page 26 / 118 20 / 46 bits 206. The arithmetic encoder can be a Huffman encoding, a range encoding, or an asymmetric numeral system (ANS) encoding.

[0039] The data flow management phase is the management of compressed data 208. In some example conventional coding systems, the compressed data is in a single bitstream. Some example neural-based coding systems use multiple streams for simultaneous coding.

[0040] A key principle of entropy coding, derived from the basic principles of information theory, is that the ideal number of bits N^ to be assigned to encode a data symbol is: Nb = -log2(p), Equation (1) where is the estimated probability of the value of the symbol to be encoded, with knowledge of p shared by the encoder and the decoder (e.g., media encoder 200 and media decoder 400). This ideal number for the average number of bits is a fractional number, but can be effectively achieved in practice using arithmetic encoding.

[0041] Symbol probabilities can vary widely and frequently depending on the specific parts of the media content. Thus, in conventional media compression, one of the most important tasks is the efficient estimation of these data symbol probabilities and how to accurately update the data symbol probabilities as they change during encoding. Once this probability estimate is made Petition 870250081169, dated 10 / 09 / 2025, page 27 / 118 21 / 46 sequentially, it can be difficult to parallelize entropy encoding in conventional video encoding processes.

[0042] The compression processes proposed for neural-based codecs are designed to work on the assumption that the data to be compressed has a certain type of probability density function (PDF) and only the parameters that define this function can vary. As an example, the PDF might be chosen to have all variables with independent Gaussian distributions with zero mean, each with a different standard deviation σ. This may not simply be an empirical assumption, since the networks learn to make it real. The PDF is typically, but not always, Gaussian.

[0043] In general, the standard deviation parameter is a scaling factor for a template distribution. For example, the Gaussian or Laplacian with zero mean and unit variance are templates, and there are scaling parameters to lengthen or shrink the template. Commonly, the standard deviation is used due to its convenience, but it can be any factor.

[0044] An important class of proposed neural-based image and video compression processes adapts the variational autoencoder architecture, where latent variables are quantized and entropy-encoded using fixed learned distributions. In some examples, this architecture has been extended by adding a smaller neural network, called hyperpriority, which uses the variables generated from the autoencoder to define the probability distributions that will be used for the Petition 870250081169, dated 10 / 09 / 2025, page 28 / 118 22 / 46 entropy encoding of these autoencoder variables. This hyperpriority network also uses quantization and entropy encoding, but with predefined (e.g., learned) PDF parameters.

[0045] This disclosure describes sliding window rate-distortion optimization techniques that address the problems introduced above. The techniques in this disclosure can enable E2E-NVCs, such as the Media Encoder 200 and Media Decoder 400, to use P-frame sequences that are comparable in length to the sequences used by conventional video encoders without increased distortion as the E2E-NVC advances through the P-frame sequence.

[0046] Figure 3 shows an example of a set of input frames 300A to 300J, covering a time sequence from time t = 0 to t = 8. The media encoder 200 sequentially encodes input frames 300A to 300J, using an image analysis neural network, to produce output frames 302A to 302J. Thus, output frame 302A corresponds to an encoded version of input frame 300A; output frame 302B corresponds to an encoded version of input frame 300B, and so on. Output frames 302A to 302J may, for example, be subsampled versions of input frames 300A to 300J. Output frames 302A to 302J may be part of a larger group of images (GOP - larger group of images) that begins with an I-frame. Output frames 302A to 302J may, for example, all be P-frames that follow the initial I-frame in the GOP. In other examples, output frame 302A may be an I-frame, while frames 302B to 302J may be P-frames or B-frames. Petition 870250081169, dated 10 / 09 / 2025, p. 29 / 118 23 / 46

[0047] The media encoder 200 can optimize the output frames 302A to 302J using a sliding window. That is, the media encoder 200 can modify the sample values ​​of any or all of the output frames 302A to 302J in order to improve the overall encoding quality across all frames. To modify the sample value, the media encoder 200 can, for example, modify a predictor or residual value for the sample, or it can modify a singular value indicative of the sample value. For example, using a window size of w=5, the media encoder 200 can determine a modification in the output frame 302A that results in an optimized amount of distortion between the output frames 302A to 302E and the input frames 300A to 300E. The media encoder 200 can then determine a modification in the output frame 302B that results in an optimized amount of distortion between the output frames 302B to 302F and the input frames 300B to 300F.Thus, in the example in Figure 3, each output frame is modified based on the next four frames that follow in time. That is, as shown in Figure 3, frames 302A to 302J are arranged in temporal encoding order, which may be the same as the display order or may be different from the display order. In other words, frame 302A is encoded before frame 302B, frame 302B is encoded before frame 302C, and so on. In this way, frame 302A can act as a reference frame for frames 302B to 302J, frame 302B can act as a reference frame for frames 302C to 302J, and so on. Other window sizes, both smaller and larger, can be used instead of five. Petition 870250081169, dated 10 / 09 / 2025, page 30 / 118 24 / 46

[0048] As described above, each of the output frames 302A to 302J are P-frames and are thus decoded based on predictions that depend on previous frames. For example, frame 302J can be predicted based on frame 302I, which is predicted based on frame 302H, which in turn is predicted based on frame 302F, and so on. By modifying output frame 302A, media encoder 200 can, for example, modify output frame 302A in a way that reduces the total distortion between output frames 302B to 302E and input frames 300B to 300E, even if such modification increases the total distortion between output frame 302A and input frame 300A. For example, media encoder 200 can modify output frame 302A in a way that improves the performance of output frame 302A as a reference frame for one or more of frames 302B to 302J.Such modifications may be for prediction information, residual information, or both, to create reference samples for one or more of the 302B to 302J frames. The 200 media encoder may, in fact, prioritize minimizing distortion between frames that are temporally later in the window, such as input frame 300E and output frame 302E at time t = 4, compared to frames that are temporally earlier in the window, such as input frame 300B and output frame 302B at time t = 1.

[0049] The media encoder 200 can modify an output frame in such a way that it increases the distortion between that output frame and its corresponding input frame, but, by doing so, the media encoder 200 can make the output frame a better predictor (i.e., reference frame). Petition 870250081169, dated 10 / 09 / 2025, page 31 / 118 25 / 46 for future frames. Thus, by modifying a current frame based on a sliding window of future frames in this way, the media encoder 200 can reduce the total distortion between input frames 300A to 300J and output frames 302A to 302J, but more significantly, by modifying a current frame based on a sliding window of future frames in this way, the media encoder 200 can achieve a more constant distortion rate, instead of a variable distortion rate, in all output frames 302A to 302J and input frames 300A to 300J.

[0050] A more detailed description of the sliding window rate-distortion optimization will now be described. To perform end-to-end neural video coding, media encoder 200 and media decoder 300 attempt to learn a sequential generative model that minimizes rate distortion compensation over any group of images (GOP), as expressed in equation 2 below. 1N£ = n+ XD(xi, M ·1-1Equation (2)

[0051] In equation (2), R represents the rate estimate for y±, which represents an entropy-encoded representation of the frame xí, with i representing the frame timestamp and N representing the GOP size. A rate estimate generally represents an estimate of the number of bits required to encode the GOP. In equation (2), D represents a measure of distortion between the original frame (xí) and its reconstruction (x'i). Thus, xí is the frame that results from encoding x± and then decoding the encoded version of xí. As the process Petition 870250081169, dated 10 / 09 / 2025, page 32 / 118 26 / 46 encoding and decoding is not lossless; xi will not perfectly match xi. The amount of difference between the original frame (xi) and the decoded frame (x'i) is called distortion and can generally be measured as a function of the peak signal-to-noise ratio, a root mean square error, or other such techniques.

[0052] In equation (2), λ represents a Lagrange multiplier to control the rate-distortion tradeoff. Rate and distortion are typically inversely proportional, so increasing rate decreases distortion and decreasing rate increases distortion. The value of λ controls this tradeoff.

[0053] The media encoder 200 and media decoder 300 can be configured to perform intracoding, which begins by encoding a still image using an image codec. This initializes the GOP and creates an I-frame. The media encoder 200 and media decoder 300 can also perform intercoding, in which, using one or more previously reconstructed, i.e., decoded, frames, the media encoder 200 and media decoder 300 encode the current frame. Intercoding exploits temporal redundancies not present in intracoding and creates P-frames. Intercoding can also include bidirectional interprediction, in which a current frame is predicted from two other frames, such as a frame preceding the current frame in display order and a frame following the current frame in display order, thus creating a B-frame. The GOPs of this disclosure can include I-frames followed by P-frames and / or B-frames. Petition 870250081169, dated 10 / 09 / 2025, p. 33 / 118 27 / 46

[0054] As presented above, when operating with traditional video codecs, GOPs are generally large, with few (if any) I-frames. In this case, codecs must solve the challenging balance of bit allocation and quality propagation across long sequences. This is achieved through complex decisions made by hand-designed encoders, such as H.266. Large GOPs are especially preferable as P-frames can typically be encoded with significantly fewer bits, resulting in smaller file sizes. E2ENVCs, however, typically perform better with much smaller GOPs, compromising performance on a relative basis.In cases where an E2E-NVC uses a large or infinite GOP (a video sequence with only one I-frame), typical video quality measures, such as peak signal-to-noise ratio (PSNR), will degrade rapidly, making it imperative to regularly insert I-frames to maintain stable video quality. In short, E2E-NVCs, unlike, for example, H.266, perform poorly in bit allocation balancing in long sequences of P-frames.

[0055] To understand the likely cause, it is important to consider how E2E-NVCs are trained. That is, when working with E2E-NVCs, it is not possible to train with the large GOP sizes that may be desirable for use in practical applications. Among other reasons, training with longer sequences often uses an impractical amount of memory and is time-consuming. As an illustration of this, H.266 would generally consider 32 frames as a small GOP, while the maximum set length Petition 870250081169, dated 10 / 09 / 2025, page 34 / 118 The popular 28 / 46 E2E-NVC Vimeo90k training data is only 7 frames. Therefore, professionals currently expect E2E-NVC to be able to infer hand-designed bit allocation rules in long sequences native to traditional video codecs when trained only on small sequences, but this is generally not achieved.

[0056] To circumvent this challenge, this disclosure introduces an encoding step in E2E-NVCs that can mimic the bit allocation decisions of traditional codecs. More specifically, this disclosure describes techniques for configuring the Media Encoder 200, using a sliding window, to perform gradient steps on each yie its quantization step size before encoding and entropy quantization. The techniques in this disclosure additionally include configuring the Media Encoder 200 to perform gradient steps on the derivative of a modified distortion-rate objective (Equation (4) below) designed to promote video quality propagation to frames outside the current window.

[0057] The techniques described in the present invention may include configuring the media encoder 200 to send a trivial number of additional bits per frame, but may result in a 20% excess improvement in the Bjontegaard-Delta rate compared to existing approaches. The additional bits may, for example, include a scaling value that can be used to scale and rescale the latent variable presentation. This could, for example, be a 16-bit floating-point value. These additional bits, however, are not essential to the other techniques described in the present invention. Petition 870250081169, dated 10 / 09 / 2025, page 35 / 118 29 / 46 techniques from this disclosure may additionally enable the use of an infinite GOP configuration, which is currently not possible without the sliding window and consideration of the modified rate-distortion target.

[0058] In the following description, the entropy-encoded representation of the frame is represented as yi, such that: yi = yi ò'í' Equation (3) where õi is a quantization step size on the scale, for example, a rounding bin size to control precision. The value of μί is a prediction of yi and [ ·] is a rounding function. The quantization step size on the scale represents an amount of quantization and therefore an amount of compression, applied to the difference between the frame and the frame prediction. For a window of length w, the media encoder 200 can be configured to minimize the following term: minimize < Equation (4) using gradient descent updates of yj and õj only in the current frame (i.e., j = 1, so that global i refers to local position J). That is, the media encoder The 200 media encoder can calculate the gradient of the loss function with respect to yj and õj. The 200 media encoder can then update these quantities using the calculated gradients towards a minimum loss value. Gradient descent Petition 870250081169, dated 10 / 09 / 2025, page 36 / 118 30 / 46 represents the optimization strategy used to train neural networks. In this example, j refers to the position within a window and i refers to the position along the entire sequence. Furthermore, it should be noted that the term above uses the function g(j) instead of λ, as in equation 2. The function g(j) represents a monotonic intrawindow frame step function that increases the importance of distortion for later frames in the sequence. That is, as described above, g(j) can achieve the purpose of prioritizing the minimization of distortion between frames that are temporally later in the window compared to frames that are temporally earlier in the window. This function allows the approach to work when considering only a small data window at any given time.

[0059] Given a frame I ew, the general encoding algorithm for a sequence of length L can be represented by Algorithm 1 below. Algorithm 1: procedure Compress(x2:L) for i = 2; i < L; i = i + 1 do Minimize yi and δi subject to Eq. (3) for xi:i+w Entropy encode yAi and δί terminate to return x'2:L terminate procedure

[0060] As mentioned, the ability to adaptively allocate bits in real time allows the described process to not require subsequent I-frames after the first I-frame, thus achieving an infinite GOP. This results in a lower bit rate with minimal degradation of Petition 870250081169, dated 10 / 09 / 2025, p. 37 / 118 31 / 46 quality and a 20% improvement in the Bjontegaard-Delta rate compared to processes evaluated under a lower GOP. [00 61] Figure 4 is a diagram of a neural image codec that can utilize the sliding window rate-distortion optimization techniques of this disclosure. The process shown in Figure 4, where neural networks for image analysis and synthesis implement the variational autoencoder and the entropy encoding of its variables is defined by the hyperpriority decoder network (used in both the transmitter and receiver). Since encoder and decoder (e.g., media encoder 200 and media decoder 400) have been used to denote autoencoder parts, in the neural network literature it is common to refer to systems traditionally called by these names as transmitter and receiver, respectively.

[0062] In Figure 4, the media encoder 200 may include image analysis neural network 402, hyperpriority encoder neural network 404, quantization process 406, entropy encoding process 408, quantization process 416, and entropy encoding process 418. The media encoder 200 may also include entropy decoding process 412, hyperpriority decoder neural network 424, entropy decoding process 422, and image synthesis neural network 426 (which may be similar to a reconstruction loop in a hybrid video encoder). Although not shown, the media decoder 400 may include structures similar to the entropy decoding process 412, hyperpriority decoder neural network 424, entropy decoding process 422, and image synthesis neural network 426. Petition 870250081169, dated 10 / 09 / 2025, page 38 / 118 32 / 46

[0063] The image analysis neural network 402 is a neural network configured to encode and / or compress the input images 401. According to the techniques of this disclosure, the image analysis neural network 402 encodes a set of input frames to generate a corresponding set of output frames. The quantization unit 416 then quantizes the set of output frames. According to the techniques of this disclosure, the media encoder 200 can determine a modification in an output frame that results in an optimized amount of distortion between the input frame set and the decoded versions of the corresponding set of output frames. The modification can be a change in a sample value of the output frame or a change in the amount of quantization applied by the quantization unit 416. The entropy encoding unit 418 can then entropy-encode the updated output frame.

[0064] The compressed data stream created by the image analysis neural network 402 is then processed by the hyperpriority encoder neural network 404. The output of the hyperpriority encoder neural network 404 is then quantized by the quantization process 406 and entropy-encoded by the entropy encoding process 408 to create bitstream 1 410. The hyperpriority encoder neural network 404 is part of the process set up to determine parameters (e.g., a standard deviation) of a PDF for the data stream created by the image analysis neural network 402. The PDF parameters can then be used to entropy-encode the data stream of the image analysis neural network 402. Petition 870250081169, dated 10 / 09 / 2025, page 39 / 118 33 / 46

[0065] The entropy encoding process itself 408 uses PDF parameters (e.g., learned parameter 414) that are learned from a plurality of images to be encoded. The entropy decoding process 412 uses the same PDF parameters to entropy decode bitstream 1 410. This entropy-decoded bitstream is then processed by the hyperpriority decoder neural network 424 to generate the PDF parameters for the specific image (e.g., input image 401) being decoded.

[0066] The PDF parameters generated by the hyperpriority decoder neural network 424 are then used by the entropy encoding process 418 to encode the data stream produced by the image analysis neural network 402 after quantization by the quantization process 416. The entropy encoding process 418 creates bitstream 2 420. Bitstream 2 420 is then entropy decoded by the entropy decoding process 422 using the same PDF parameters as the hyperpriority decoder neural network 424. After entropy decoding, the image synthesis neural network 426 processes the decoded data to produce the output images 428.

[0067] Figure 5 is a flowchart illustrating an example process for encoding a set of input frames, according to the techniques of this disclosure. Although described in relation to the Video Encoder 200 (Figures 1 and 4), it should be understood that other devices can be configured to perform a process similar to that in Figure 5.

[0068] In this example, the media encoder Petition 870250081169, dated 10 / 09 / 2025, page 40 / 118 34 / 46 200 encodes a set of input frames from the video data with an image analysis neural network to generate a corresponding set of output frames (500). The output frame set includes a first output frame and additional output frames, and the first output frame temporally precedes the additional output frames. The first output frame may, for example, be an interpredicted frame, and the additional output frames may also all be interpredicted frames. The first output frame and the additional output frames may belong to a group of images that includes only one intracoded frame, and the first output frame and the additional output frames may all follow the single intracoded frame. The output frame set may also include a second output frame that temporally follows the first output frame, and the media encoder 200 may predict the second output frame based on the first output frame.In some examples, the output frameset might consist of, for instance, five or six total output frames, although more or fewer output frames could also be used. The output frameset might, for example, include subsampled versions of the input frameset.

[0069] Media encoder 200 determines a modification in the first output frame that results in an optimized amount of distortion between the input frameset and the decoded versions of the corresponding output frameset (502). To determine the modification in the first output frame that results in the optimized amount of distortion between the input frameset and the decoded versions of the set Petition 870250081169, dated 10 / 09 / 2025, page 41 / 118 35 / 46 corresponding output frames, the media encoder 200 can determine a modified scaled quantization step size for the first output frame.

[0070] To determine the modification in the first output frame that results in the optimized amount of distortion between the input frame set and the decoded versions of the corresponding output frame set, the media encoder 200 can, for example, be configured to determine, for a first decoded version of a first output frame from the output frame set and a corresponding first input frame, a first amount of distortion; determine, for a second decoded version of a second output frame from the output frame set and a corresponding second input frame, a second amount of distortion; determine a first weighting factor for the first amount of distortion based on a temporal position of the first decoded version of the first output frame;Determine a second weighting factor for the second amount of distortion based on a temporal position of the second decoded version of the second output frame, where the first weighting factor is different from the second weighting factor; and determine the optimized amount of distortion based on the first weighting factor and the second weighting factor. In this example, the second output frame may temporally follow the first output frame, and the second weighting factor may be greater than the first weighting factor.

[0071] Media encoder 200 updates the first output frame based on the determined modification. Petition 870250081169, dated 10 / 09 / 2025, page 42 / 118 36 / 46 (504). The media encoder 200 can, for example, update the first output frame by changing a sample value of at least one sample of the first output frame. In some cases, updating the first output frame may increase an amount of distortion between the first output frame and a decoded version of the corresponding input frame, in order to reduce distortion between decoded versions of later output frames and corresponding input frames. In some examples, the media encoder 200 updates the first output frame by changing a scaled quantization step size for the first output frame to the modified scaled quantization step size.

[0072] Media encoder 200 outputs a bitstream that includes the first updated output frame (506). Media encoder 200 can, for example, output the bitstream for storage or transmission.

[0073] The following numbered clauses illustrate one or more aspects of the techniques and devices described in this disclosure.

[0074] Clause 1: A method for encoding video data, wherein the method comprises: encoding a set of input frames of video data with an image analysis neural network to generate a corresponding set of output frames, wherein the set of output frames includes a first output frame and additional output frames, wherein the first output frame temporally precedes the additional output frames; determining a modification to the first output frame that results in an optimized amount of distortion between the Petition 870250081169, dated 10 / 09 / 2025, p. 43 / 118 37 / 46 set of input frames and the decoded versions of the corresponding set of output frames; update the first output frame based on the determined modification; and emit a bitstream that includes the updated first output frame.

[0075] Clause 2: The method of clause 1, where the output frameset comprises subsampled versions of the input frameset.

[0076] Clause 3: The method of clause 1 or 2, where the update of the first output frame comprises changing a sample value of at least one sample of the first output frame.

[0077] Clause 4: The method of any of clauses 1 to 3, wherein updating the first output frame increases an amount of distortion between the first output frame and a decoded version of the corresponding input frame.

[0078] Clause 5: The method of any of clauses 1 to 4, wherein the set of output frames includes a second output frame that temporally follows the first output frame, wherein encoding the set of input video data frames with the image analysis neural network to generate the corresponding set of output frames comprises predicting the second output frame based on a decoded version of the first output frame.

[0079] Clause 6: The method of any of clauses 1 to 5, where the first output frame comprises an interpredicted frame and the additional output frames are all interpredicted frames.

[0080] Clause 7: The method of any of Petition 870250081169, dated 10 / 09 / 2025, p. 44 / 118 38 / 46 clauses 1 to 6, where the first output frame and the additional output frames belong to a group of images that includes only one intracoded frame and where the first output frame and the additional output frames are all interpredicted frames.

[0081] Clause 8: The method of any of clauses 1 to 7, wherein the determination of the modification in the first output frame that results in the optimized amount of distortion between the input frame set and the decoded versions of the corresponding output frame set comprises: determining, for a first decoded version of a first output frame from the output frame set and a corresponding first input frame, a first amount of distortion; determining, for a second decoded version of a second output frame from the output frame set and a corresponding second input frame, a second amount of distortion; determining a first weighting factor for the first amount of distortion based on a temporal position of the first decoded version of the first output frame;Determine a second weighting factor for the second amount of distortion based on a time position of the second decoded version of the second output frame, where the first weighting factor is different from the second weighting factor; determine the optimized amount of distortion based on the first weighting factor and the second weighting factor.

[0082] Clause 9: The method of clause 8, in which the second output frame temporally follows the first output frame and the second weighting factor is greater than Petition 870250081169, dated 10 / 09 / 2025, page 45 / 118 39 / 46 which is the first weighting factor.

[0083] Clause 10: The method of any of clauses 1 to 9, wherein the set of output frames consists of the first output frame and four additional output frames.

[0084] Clause 11: The method of any of clauses 1 to 9, where the set of output frames consists of the first output frame and five additional output frames.

[0085] Clause 12: The method of any of clauses 1 to 11, wherein the determination of the modification in the first output frame that results in the optimized amount of distortion between the input frameset and the decoded versions of the corresponding output frameset comprises determining a modified scaled quantization step size for the first output frame.

[0086] Clause 13: A device for encoding video data, wherein the device comprises: a memory configured to store video data; one or more processors implemented in a circuit array and configured to: encode a set of input frames of video data with an image analysis neural network to generate a corresponding set of output frames, wherein the set of output frames includes a first output frame and additional output frames, wherein the first output frame temporally precedes the additional output frames; determine a modification to the first output frame that results in an optimized amount of distortion between the set of input frames. Petition 870250081169, dated 10 / 09 / 2025, page 46 / 118 40 / 46 and the decoded versions of the corresponding set of output frames; update the first output frame based on the determined modification; and emit a bitstream that includes the updated first output frame.

[0087] Clause 14: The device of clause 13, where the output frameset comprises subsampled versions of the input frameset.

[0088] Clause 15: The device of either clause 13 and 14, where, to update the first output frame, one or more processors are additionally configured to change a sample value of at least one sample of the first output frame.

[0089] Clause 16: The device of any of clauses 13 to 15, in which updating the first output frame increases an amount of distortion between the first output frame and a decoded version of the corresponding input frame.

[0090] Clause 17: The device of any of clauses 13 to 16, wherein the set of output frames includes a second output frame that temporally follows the first output frame, wherein to encode the set of input video data frames with the image analysis neural network to generate the corresponding set of output frames, one or more processors are additionally configured to predict the second output frame based on a decoded version of the first output frames.

[0091] Clause 18: The device of any of clauses 13 to 17, wherein the first output frame comprises an interpredicted frame and the output frames Petition 870250081169, dated 10 / 09 / 2025, p. 47 / 118 41 / 46 additional figures are all interpreted frames.

[0092] Clause 19: The device of any of clauses 13 to 18, wherein the first output frame and the additional output frames belong to a group of images that includes only one intracoded frame and wherein the first output frame and the additional output frames are all interpredicted frames.

[0093] Clause 21: The device of any of clauses 13 to 19, wherein, to determine the modification in the first output frame that results in the optimized amount of distortion between the input frame set and the decoded versions of the corresponding output frame set, one or more processors are additionally configured to: determine for a first decoded version of a first output frame of the output frame set and a corresponding first input frame, a first amount of distortion; determine, for a second decoded version of a second output frame of the output frame set and a corresponding second input frame, a second amount of distortion; determine a first weighting factor for the first amount of distortion based on a temporal position of the first decoded version of the first output frame;Determine a second weighting factor for the second amount of distortion based on a time position of the second decoded version of the second output frame, where the first weighting factor is different from the second weighting factor; determine the optimized amount of distortion based on the first weighting factor and the second weighting factor. Petition 870250081169, dated 10 / 09 / 2025, page 48 / 118 42 / 46

[0094] Clause 22: The provision of clause 21, where the second output frame follows the first output frame temporally and the second weighting factor is greater than the first weighting factor.

[0095] Clause 23: The method of any of clauses 13 to 22, wherein the set of output frames consists of the first output frame and four additional output frames.

[0096] Clause 24: The device of any of clauses 13 to 22, where the set of output frames consists of the first output frame and five additional output frames.

[0097] Clause 25: The device of any of clauses 13 to 24, wherein, to determine the modification in the first output frame that results in the optimized amount of distortion between the input frame set and the decoded versions of the corresponding output frame set, one or more processors are additionally configured to determine a modified scaled quantization step size for the first output frame.

[0098] Clause 26: The device of any of clauses 13 to 25, wherein the device comprises one or more of a camera, a computer or a mobile device.

[0099] Clause 27: The device of any of the clauses 13 to 26, which additionally comprises: a camera configured to capture video data.

[00100] Clause 28: A computer-readable storage medium that stores instructions which, when executed by one or more processors, cause the one Petition 870250081169, dated 10 / 09 / 2025, page 49 / 118 43 / 46 or more processors: encode a set of input frames of video data with an image analysis neural network to generate a corresponding set of output frames, wherein the output frame set includes a first output frame and additional output frames, wherein the first output frame temporally precedes the additional output frames; determine a modification to the first output frame that results in an optimized amount of distortion between the input frame set and the decoded versions of the corresponding output frame set; update the first output frame based on the determined modification; and output a bitstream that includes the updated first output frame.

[00101] It should be recognized that, depending on the example, certain actions or events of any of the techniques described in the present invention may be performed in a different sequence, may be added, combined or completely omitted (for example, not all actions or events described are necessary for the practice of the techniques). Furthermore, in certain examples, the actions or events may be performed simultaneously, for example, through multi-threaded processing, interrupt processing or multiple processors, instead of sequentially.

[00102] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored or transmitted as one or more instructions or code in a computer-readable medium and executed by a processing unit. Petition 870250081169, dated 10 / 09 / 2025, page 50 / 118 44 / 46 based on hardware. Computer-readable media may include computer-readable storage media, which correspond to a tangible medium, such as data storage media, or communication media, including any medium that facilitates the transfer of a computer program from one place to another, for example, according to a communication protocol. In this way, computer-readable media may generally correspond to (1) tangible computer-readable storage media that are non-transient or (2) a communication medium, such as a signal or a carrier wave. Data storage media may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for the implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[00103] By way of example, and not limitation, such computer-readable storage media may comprise random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory or any other media that can be used to store the desired program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection is suitably Petition 870250081169, dated 10 / 09 / 2025, page 51 / 118 45 / 46 termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave will be included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but instead refer to tangible non-transient storage media.As used in the present invention, disks (disk and disc) include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks typically reproduce data magnetically, while discs reproduce data optically by means of lasers. Combinations of the above items should also be included within the scope of computer-readable media.

[00104] The instructions may be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent discrete or integrated logic circuitry. Consequently, the terms processor and processing circuitry, as used in the present invention, may refer to any of the foregoing structures or any other suitable structure. Petition 870250081169, dated 10 / 09 / 2025, page 52 / 118 46 / 46 for the implementation of the techniques described in the present invention. Furthermore, in some aspects, the functionality described in the present invention can be provided in dedicated hardware and / or software modules configured for encoding and decoding, or can be incorporated into a combined codec. Additionally, the techniques can be fully implemented in one or more circuits or logic elements.

[00105] The techniques in this disclosure can be implemented in a wide variety of devices or appliances, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require implementation by different hardware units. Instead, as described above, multiple units can be combined into a codec hardware unit or provided by a collection of interoperable hardware units, including one or more processors, as described above, in conjunction with suitable software and / or firmware.

[00106] Several examples have been described. These and other examples are within the scope of the following claims. Petition 870250081169, dated 10 / 09 / 2025, page 53 / 118

Claims

1 / 8 CLAIMS 1. A method for encoding video data characterized by comprising: encoding a set of input frames of video data with an image analysis neural network to generate a corresponding set of output frames, wherein the output frame set includes a first output frame and additional output frames, wherein the first output frame temporally precedes the additional output frames; determining a modification in the first output frame that results in an optimized amount of distortion between the input frame set and the decoded versions of the corresponding output frame set; updating the first output frame based on the determined modification; and emitting a bitstream that includes the updated first output frame.

2. A method according to claim 1, characterized in that the output frameset comprises subsampled versions of the input frameset.

3. Method, according to claim 1, characterized in that updating the first output frame comprises changing a sample value of at least one sample from the first output frame.

4. A method according to claim 1, characterized in that updating the first output frame increases the amount of distortion between the first output frame and a decoded version of the corresponding input frame.

5. Method according to claim 1, Petition 870250081169, dated 10 / 09 / 2025, pp. 101 / 118 2 / 8 characterized in that the set of output frames includes a second output frame that temporally follows the first output frame, wherein encoding the set of input video data frames with the image analysis neural network to generate the corresponding set of output frames comprises predicting the second output frame based on a decoded version of the first output frame.

6. Method according to claim 1, characterized in that the first output frame comprises an interpredicted frame and the additional output frames are all interpredicted frames.

7. Method according to claim 1, characterized in that the first output frame and the additional output frames belong to a group of images that includes only one intracoded frame and in which the first output frame and the additional output frames are all interpreted frames.

8. Method, according to claim 1, characterized by the determination of the modification in the first output frame that results in the optimized amount of distortion between the input frameset and the decoded versions of the corresponding output frameset comprising: determining, for a first decoded version of a first output frame from the output frameset and a corresponding first input frame, a first amount of distortion; determining, for a second decoded version of a second output frame from the output frameset and a corresponding second input frame, a second amount of distortion. Petition 870250081169, dated 10 / 09 / 2025, p.102 / 118 3 / 8 amount of distortion; determine a first weighting factor for the first amount of distortion based on a time position of the first decoded version of the first output frame; determine a second weighting factor for the second amount of distortion based on a time position of the second decoded version of the second output frame, where the first weighting factor is different from the second weighting factor; determine the optimized amount of distortion based on the first weighting factor and the second weighting factor.

9. Method, according to claim 8, characterized in that the second output frame temporally follows the first output frame, and the second weighting factor is greater than the first weighting factor.

10. Method according to claim 1, characterized in that the set of output frames consists of the first output frame and four additional output frames.

11. Method according to claim 1, characterized in that the set of output frames consists of the first output frame and five additional output frames.

12. Method, according to claim 1, characterized by determining the modification in the first output frame that results in the optimized amount of distortion between the input frameset and the decoded versions of the corresponding output frameset. Petition 870250081169, dated 10 / 09 / 2025, pp. 103 / 118 4 / 8 comprising determining a modified scaled quantization step size for the first output frame.

13. Device for encoding video data characterized by comprising: a memory configured to store video data; one or more processors implemented in the circuit set and configured to: encode a set of input frames of video data with an image analysis neural network to generate a corresponding set of output frames, wherein the set of output frames includes a first output frame and additional output frames, wherein the first output frame temporally precedes the additional output frames; determine a modification in the first output frame that results in an optimized amount of distortion between the input frame set and the decoded versions of the corresponding set of output frames; update the first output frame based on the determined modification; and output a bitstream that includes the updated first output frame.

14. Device according to claim 13, characterized in that the output frameset comprises subsampled versions of the input frameset.

15. Device according to claim 13, characterized in that, to update the first output frame, one or more processors are additionally configured to change a sample value of at least one sample of the first output frame. Petition 870250081169, dated 10 / 09 / 2025, pp. 104 / 118 5 / 8 16. Device according to claim 13, characterized in that updating the first output frame increases the amount of distortion between the first output frame and a decoded version of the corresponding input frame.

17. Device according to claim 13, characterized in that the set of output frames includes a second output frame that temporally follows the first output frame, wherein to encode the input frame set of video data with the image analysis neural network to generate the corresponding set of output frames, one or more processors are additionally configured to predict the second output frame based on a decoded version of the first output frames.

18. Device according to claim 13, characterized in that the first output frame comprises an interpredicted frame and the additional output frames are all interpredicted frames.

19. Device according to claim 13, characterized in that the first output frame and the additional output frames belong to a group of images that includes only one intracoded frame and in which the first output frame and the additional output frames are all interpreted frames.

20. Device, according to claim 13, characterized by, for determining the modification in the first output frame that results in the optimized amount of distortion between the input frame set and the decoded versions of the corresponding output frame set, the one or more processors being additionally Petition 870250081169, dated 10 / 09 / 2025, p.105 / 118 6 / 8 configured to: determine, for a first decoded version of a first output frame from the output frameset and a corresponding first input frame, a first amount of distortion; determine, for a second decoded version of a second output frame from the output frameset and a corresponding second input frame, a second amount of distortion; determine a first weighting factor for the first amount of distortion based on a time position of the first decoded version of the first output frame; determine a second weighting factor for the second amount of distortion based on a time position of the second decoded version of the second output frame, where the first weighting factor is different from the second weighting factor; determine the optimized amount of distortion based on the first weighting factor and the second weighting factor.

21. Device according to claim 20, characterized in that the second output frame temporally follows the first output frame, and the second weighting factor is greater than the first weighting factor.

22. Device according to claim 13, characterized in that the set of output frames consists of the first output frame and four additional output frames.

23. Device according to claim 13, Petition 870250081169, dated 10 / 09 / 2025, pp. 106 / 118 7 / 8 characterized in that the set of output frames consists of the first output frame and five additional output frames.

24. Device according to claim 13, characterized in that, to determine the modification in the first output frame that results in the optimized amount of distortion between the input frame set and the decoded versions of the corresponding output frame set, one or more processors are additionally configured to determine a modified scaled quantization step size for the first output frame.

25. Device according to claim 13, characterized by comprising one or more of a camera, a computer or a mobile device.

26. Device according to claim 13, characterized by further comprising: a camera configured to capture video data.

27. A computer-readable storage medium characterized by storing instructions that, when executed by one or more processors, cause the one or more processors to: encode a set of input frames of video data with an image analysis neural network to generate a corresponding set of output frames, wherein the set of output frames includes a first output frame and additional output frames, wherein the first output frame temporally precedes the additional output frames; determine a modification in the first output frame that results in an optimized amount of distortion between the set of input frames and the decoded versions of the corresponding set of output frames; update the first output frame based on the determined modification; and output a bitstream that includes the updated first output frame.108 / 118.

Images