Video Coding Using Separate Learning and Filtering Paths

Abstract

Separate pathways for filtering and machine learning are introduced within a video decoder. A first pathway filters a first copy of a reconstructed frame to produce a filtered frame, which is included in an output video stream for display. A second pathway processes a second copy of the reconstructed frame using a learning model, such as for training and / or inference. The first pathway and the second pathway are introduced after the reconstruction stage of an encoder or decoder. Thus, the inputs to each of the first pathway and the second pathway are generated without the use of non-injective functions, and while the first pathway includes at least one non-injective function, the second pathway does not. Thus, training the learning model using the second copy of the reconstructed frame results in a greater upper limit on classification accuracy than training the learning model using the filtered frame.

Description

Background Art

[0001] A digital video stream can represent a video using a sequence of frames or still images. Digital video can be used for a variety of applications, including, for example, video conferencing, high-definition video entertainment, video advertising, or user-generated video sharing. A digital video stream can contain a large amount of data and consume a large amount of computing or communication resources of a computing device to process, transmit, or store video data. Various methods (including encoding or decoding techniques) have been proposed to reduce the amount of data in a video stream. Summary of the invention

[0002] A decoder according to an embodiment of the present disclosure includes an entropy decoding stage, a dequantization stage, an inverse transform stage, a reconstruction stage, a first post-reconstruction path, and a second post-reconstruction path. The entropy decoding stage entropy decodes syntax elements representing a coded video frame from a coded bitstream to generate quantized transform coefficients. The dequantization stage dequantizes the quantized transform coefficients to generate transform coefficients. The inverse transform stage inversely transforms the transform coefficients to generate prediction residuals. The reconstruction stage reconstructs the prediction residuals to generate a reconstructed frame. The first post-reconstruction path filters a first copy of the reconstructed frame using one or more filtering techniques to generate a filtered frame, wherein the filtered frame is included in an output video stream for display on a user device. The second post-reconstruction path trains a learning model using a second copy of the reconstructed frame. Compared with training the learning model using a filtered frame, training the learning model using a second copy of the reconstructed frame results in a larger upper limit on the classification accuracy of the learning model.

[0003] The method according to an embodiment of the present disclosure includes dequantizing quantized transform coefficients representing encoded video data to produce transform coefficients. The method further includes inverse transforming the transform coefficients to produce prediction residuals. The method further includes reconstructing the prediction residuals to produce a reconstructed frame. The method further includes filtering a first copy of the reconstructed frame to produce a filtered frame. The method further includes processing a second copy of the reconstructed frame using a learning model to identify video content.

[0004] An integrated circuit according to an embodiment of the present disclosure includes a processor for executing instructions. The instructions include decoding encoded video data from an encoded bitstream to generate a reconstructed frame. The instructions further include processing a first copy of the reconstructed frame on a first decoding path using an in-loop filter to generate an output video stream for display on a user device. The instructions further include processing a second copy of the reconstructed frame on a second decoding path using a learning model to identify video content.

[0005] These and other aspects of the disclosure are disclosed in the following detailed description of the embodiments, the appended claims and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The description herein refers to the drawings described below, wherein like reference numerals refer to like parts throughout the several views.

[0007] Figure 1 is a schematic diagram of an example of a video encoding and decoding system.

[0008] Figure 2 is a block diagram of an example of a computing device that may implement a transmitting station or a receiving station.

[0009] Figure 3 is a diagram of an example of a video stream to be encoded and then decoded.

[0010] Figure 4 is a block diagram of an example of an encoder according to an embodiment of the present disclosure.

[0011] Figure 5 is a block diagram of an example of a decoder according to an embodiment of the present disclosure.

[0012] Figure 6 is a block diagram of an example of an encoder including separate learning and filtering paths.

[0013] Figure 7 is a block diagram of an example of a decoder including separate learning and filtering paths.

[0014] Figure 8 is a flow chart of an example of a technique for video coding using separate learning and filtering paths.

[0015] Fig. 9 is a flow chart of an example of a technique for training and inference using a learned model for video coding. DETAILED DESCRIPTION

[0016] Video compression schemes may include breaking down a corresponding image or frame into smaller parts (such as blocks) and generating a compressed bitstream using a technique that limits the information included in the corresponding blocks in the output. The compressed bitstream can be decoded to recreate the source image from the limited information. Typical video compression and decompression techniques use many tools to improve coding efficiency, which can be measured by both rate and distortion. Many of these tools use injective functions with a one-to-one mapping between input and output. Therefore, the input of the injective function can be identified and recovered from the output of this injective function. However, some of those tools use non-injective functions with a one-to-many mapping between input and output. The use of non-injective functions can result in some information loss.

[0017] For example, a typical decoder processes a compressed bitstream through a series of decoding stages and ultimately sends the output of the last decoding stage in an output video stream. Each decoding stage applies a function to the input it receives from an earlier decoding stage in the series and produces an output for use by a later decoding stage in the series. Many of these decoding stages (e.g., the entropy decoding stage, the dequantization stage, the inverse transform stage, the prediction stage, and the reconstruction stage) use only injective functions; however, some of these decoding stages (e.g., the filtering stage) use non-injective functions. The use of non-injective functions causes the output video stream to no longer have a one-to-one mapping with the compressed bitstream.

[0018] In some cases, the encoder or decoder may include machine learning aspects, such as for training a learning model to identify video content and / or for reasoning about video content. A common machine learning approach in video coding is to use the output of a decoding pipeline as input to a learning model. Thus, during the decoding process, the output of the last decoding stage (e.g., filtering) is used as input to the learning model. Similarly, during the encoding process, the output of the last reconstruction path stage (e.g., filtering) can be used as input to the learning model.

[0019] However, this approach is flawed because using non-injective functions for video coding can result in information loss. For example, filtering techniques performed during the encoding or decoding process superficially process video data for human viewing purposes, such as by reducing blocking artifacts within video frames to improve the overall appearance of the video. The changes made by filtering result in information loss, which can negatively impact the performance of the learned model. For example, the learned model may misclassify the content of a video output through filtering due to the information loss caused by filtering. This misclassification can impair or limit the accuracy of machine learning.

[0020] Therefore, it is conceivable that the machine learning aspects of an encoder or decoder could benefit from the removal of non-injective functions (e.g., filter stages) in the encoder or decoder. However, those non-injective functions may still be important to the encoding or decoding process. For example, a typical encoder or decoder may cache the output of a filter stage for use in prediction of future frames. Therefore, it is not feasible to remove non-injective functions from the encoder and decoder.

[0021] Embodiments of the present disclosure solve problems such as using separate paths for filtering and for machine learning in a video encoder. A first path filters a first copy of a reconstructed frame to produce a filtered frame that is included in an output video stream for display on a user device. A second path processes a second copy of the reconstructed frame using a learning model (e.g., for training and / or reasoning). The first path and the second path are introduced after the reconstruction stage of an encoder or decoder. Therefore, the inputs to each of the first path and the second path are generated without using a non-injective function, and although the first path includes at least one non-injective function, the second path does not.

[0022] Therefore, training the learning model using the second copy of the reconstructed frame results in a greater upper limit on the classification accuracy of the learning model than training the learning model using the filtered frame. Therefore, training the learning model using the second copy of the reconstructed frame improves or at least maintains the upper limit of the performance of the learning model at that time. Separating the learning path and the filtering path further benefits the machine learning aspect of the video encoder because the learning path avoids the computational complexity associated with filtering video data.

[0023] Further details of techniques for video coding using separate learning and filtering paths are described herein, first with reference to systems in which they may be implemented. Figure 1 1 is a schematic diagram of a video encoding and decoding system 100. The transmitting station 102 may be, for example, a computer having an internal configuration of hardware, such as Figure 2 However, other implementations of the sending station 102 are possible. For example, the processing of the sending station 102 can be distributed among multiple devices.

[0024] The network 104 can connect the sending station 102 and the receiving station 106 for encoding and decoding the video stream. Specifically, the video stream can be encoded in the sending station 102, and the encoded video stream can be decoded in the receiving station 106. The network 104 can be, for example, the Internet. The network 104 can also be a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), a cellular telephone network, or any other means of transferring the video stream from the sending station 102 to the receiving station 106 in this example.

[0025] In one example, receiving station 106 may be a computer having an internal configuration of hardware, such as in Figure 2 However, other suitable implementations of the receiving station 106 are possible. For example, the processing of the receiving station 106 can be distributed among multiple devices.

[0026] Other embodiments of the video encoding and decoding system 100 are possible. For example, one embodiment can omit the network 104. In another embodiment, the video stream can be encoded and then stored for transmission to a receiving station 106 or any other device with memory at a later time. In one embodiment, the receiving station 106 receives the encoded video stream (e.g., via the network 104, a computer bus and / or some communication path) and stores the video stream for later decoding. In an example embodiment, a real-time transport protocol (RTP) is used to send the encoded video over the network 104. In another embodiment, a transport protocol other than RTP can be used, for example, a video streaming protocol based on the hypertext transfer protocol (HTTP).

[0027] For example, when used in a video conferencing system, the sending station 102 and / or the receiving station 106 may include the ability to both encode and decode video streams as described below. For example, the receiving station 106 may be a video conference participant who receives an encoded video bitstream from a video conference server (e.g., the sending station 102) for decoding and viewing, and further encodes his or her own video bitstream and sends it to the video conference server for decoding and viewing by other participants.

[0028] In some embodiments, the video encoding and decoding system 100 may alternatively be used to encode and decode data other than video data. For example, the video encoding and decoding system 100 may be used to process image data. The image data may include data blocks from an image. In such an embodiment, the sending station 102 may be used to encode the image data, and the receiving station 106 may be used to decode the image data.

[0029] Alternatively, receiving station 106 may represent a computing device that stores encoded image data for later use, such as after receiving the encoded or pre-encoded image data from sending station 102. As yet another alternative, sending station 102 may represent a computing device that decodes the image data, such as before sending the decoded image data to receiving station 106 for display.

[0030] Figure 2 is a block diagram of an example of a computing device 200 that can implement a sending station or a receiving station. For example, the computing device 200 can implement Figure 1 The computing device 200 may be in the form of a computing system including multiple computing devices, or in the form of a computing device, such as a mobile phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, etc.

[0031] The processor 202 in the computing device 200 may be a conventional central processing unit. Alternatively, the processor 202 may be any other type of device, or multiple devices now existing or hereafter developed that are capable of manipulating or processing information. For example, although the disclosed embodiments can be practiced with one processor (e.g., the processor 202) as shown, advantages in speed and efficiency can be achieved by using more than one processor.

[0032] In an embodiment, the memory 204 in the computing device 200 can be a read-only memory (ROM) device or a random access memory (RAM) device. However, any other suitable type of storage device can be used as the memory 204. The memory 204 can include code and data 206 accessed by the processor 202 using the bus 212. The memory 204 can further include an operating system 208 and application programs 210, which include at least one program that allows the processor 202 to perform the techniques described herein. For example, the application programs 210 can include application 1 to application N, which further include a video and / or image coding application that performs the techniques described herein.

[0033] The computing device 200 can also include auxiliary storage 214, which can be, for example, a memory card used with a mobile computing device. Because a video communication session can contain a large amount of information, it can be stored in full or in part in the auxiliary storage 214 and loaded into the memory 204 as needed for processing.

[0034] The computing device 200 can also include one or more output devices, such as a display 218. In one example, the display 218 can be a touch-sensitive display that combines a display with a touch-sensitive element operable to sense touch input. The display 218 can be coupled to the processor 202 via the bus 212. In addition to the display 218 or as an alternative to the display 218, other output devices that allow a user to program or otherwise use the computing device 200 can be provided. When the output device is or includes a display, the display can be implemented in various ways, including by a liquid crystal display (LCD), a cathode ray tube (CRT) display, or a light emitting diode (LED) display, such as an organic LED (OLED) display.

[0035] The computing device 200 can also include or be in communication with an image sensing device 220, such as a camera or any other image sensing device 220 now known or hereafter developed that is capable of sensing an image, such as an image of a user operating the computing device 200. The image sensing device 220 can be positioned so that it is oriented toward the user operating the computing device 200. In one example, the position and optical axis of the image sensing device 220 can be configured so that the field of view includes an area directly adjacent to and visible to the display 218.

[0036] The computing device 200 can also include or be in communication with a sound sensing device 222, such as a microphone or any other sound sensing device now known or hereafter developed that can sense sounds in the vicinity of the computing device 200. The sound sensing device 222 can be positioned so that it is oriented toward a user operating the computing device 200, and can be configured to receive sounds, such as voice or other utterances, made by the user while the user is operating the computing device 200.

[0037] although Figure 2 The processor 202 and memory 204 of the computing device 200 are depicted as being integrated into one unit, but other configurations can be utilized. The operations of the processor 202 can be distributed across multiple machines (where individual machines may have one or more processors) that can be coupled directly or across a local area network or other network. The memory 204 can be distributed across multiple machines such as network-based storage or storage in multiple machines that perform the operations of the computing device 200.

[0038] Although depicted here as one bus, the bus 212 of the computing device 200 can be comprised of multiple buses. Additionally, the auxiliary storage 214 can be directly coupled to other components of the computing device 200 or can be accessed via a network and can include one integrated unit (such as a memory card) or multiple units (such as multiple memory cards). The computing device 200 can therefore be implemented in a wide variety of configurations.

[0039] Figure 3 300 is a diagram of an example of a video stream 300 to be encoded and subsequently decoded. The video stream 300 includes a video sequence 302. At the next level, the video sequence 302 includes a plurality of adjacent frames 304. Although three frames are described as adjacent frames 304, the video sequence 302 can include any number of adjacent frames 304. The adjacent frames 304 can then be further subdivided into individual frames, such as frames 306.

[0040] At the next level, the frame 306 can be divided into a series of planes or segments 308. For example, the segments 308 can be subsets of the frame that allow for parallel processing. The segments 308 can also be subsets of the frame that can separate the video data into separate colors. For example, a frame 306 of color video data can include a luma plane and two chroma planes. The segments 308 can be sampled at different resolutions.

[0041] Regardless of whether the frame 306 is divided into segments 308, the frame 306 can be further subdivided into blocks 310, which can contain data corresponding to, for example, 16×16 pixels in the frame 306. The blocks 310 can also be arranged to include data from one or more segments 308 of pixel data. The blocks 310 can also have any other suitable size, such as 4×4 pixels, 8×8 pixels, 16×8 pixels, 8×16 pixels, 16×16 pixels, or larger. Unless otherwise specified, the terms block and macroblock are used interchangeably herein.

[0042] Figure 4 4 is a block diagram of an example of encoder 400. As described above, encoder 400 can be implemented in transmission station 102, such as by providing a computer software program stored in a memory, such as memory 204. The computer software program can include machine instructions that, when executed by a processor, such as processor 202, cause transmission station 102 to transmit the data in a manner that is consistent with the embodiment of the present invention. Figure 4 The video data is encoded in the manner described in . The encoder 400 can also be implemented as dedicated hardware, for example, included in the sending station 102. In a particularly preferred embodiment, the encoder 400 is a hardware encoder.

[0043] The encoder 400 has the following stages that perform various functions in a forward path (shown by solid connecting lines) to produce an encoded or compressed bitstream 420 using the video stream 300 as input: an intra / inter prediction stage 402, a transform stage 404, a quantization stage 406, and an entropy coding stage 408. The encoder 400 may also include a reconstruction path (shown by a dashed connecting line) to reconstruct frames for encoding future blocks. Figure 4 4 , the encoder 400 has the following stages that perform various functions in the reconstruction path: a dequantization stage 410, an inverse transform stage 412, a reconstruction stage 414, and a loop filter stage 416. Other structural variations of the encoder 400 can be used to encode the video stream 300.

[0044] When presenting the video stream 300 to encode corresponding adjacent frames 304, such as frame 306, the frame 306 can be processed in units of blocks. At the intra / inter prediction stage 402, the corresponding blocks can be encoded using intra prediction (also referred to as intra prediction) or inter prediction (also referred to as inter prediction). In either case, a prediction block can be formed. In the case of intra prediction, the prediction block can be formed from samples in the current frame that have been previously encoded and reconstructed. In the case of inter prediction, the prediction block can be formed from samples in one or more previously constructed reference frames.

[0045] Next, the prediction block can be subtracted from the current block at the intra / inter prediction stage 402 to produce a residual block (also referred to as a residual). A transform stage 404 transforms the residual into transform coefficients, for example, in the frequency domain, using a block-based transform. A quantization stage 406 uses a quantizer value or quantization level to convert the transform coefficients into discrete quantum values, which are referred to as quantized transform coefficients. For example, the transform coefficients may be divided by a quantizer value and truncated.

[0046] The quantized transform coefficients are then entropy encoded by an entropy encoding stage 408. The entropy encoded coefficients are then output to a compressed bitstream 420 along with other information used to decode the block (which may include, for example, syntax elements such as those used to indicate the prediction type used, the transform type, motion vectors, quantizer values, etc.). The compressed bitstream 420 may be formatted using various techniques such as variable length coding (VLC) or arithmetic coding. The compressed bitstream 420 can also be referred to as an encoded video stream or an encoded video bitstream, and these terms will be used interchangeably herein.

[0047] The reconstruction path (shown by the dashed connecting line) can be used to ensure that the encoder 400 and (hereinafter referred to as Figure 5 The decoder 500 described above uses the same reference frame to decode the compressed bitstream 420. The reconstruction path performs similar functions to those performed during the decoding process (see below). Figure 5 The functions described above include dequantizing the quantized transform coefficients at a dequantization stage 410 and inverse transforming the dequantized transform coefficients at an inverse transform stage 412 to produce a derivative residual block (also referred to as a derivative residual).

[0048] At reconstruction stage 414, the prediction block predicted at intra / inter prediction stage 402 can be added to the derivative residual to create a reconstructed block. Loop filtering stage 416 can apply an in-loop filter or other filter to the reconstructed block to reduce distortion such as blocking artifacts. Examples of filters include, but are not limited to: a deblocking filter as in H.264, VP9, ​​H.265 (HEVC), and AV1; a constrained directional enhancement filter as in AV1; and a loop recovery filter as in AV1.

[0049] Other variations of encoder 400 can be used to encode compressed bitstream 420. In some embodiments, a non-transform based encoder can directly quantize the residual signal without transform stage 404 for certain blocks or frames. In some embodiments, the encoder can combine quantization stage 406 and dequantization stage 410 into a common stage. In some embodiments, encoder 400 can have separate learning and filtering paths. Figure 6 Embodiments and examples of encoders with separate learning and filtering paths are shown and described.

[0050] Figure 5 2 is a block diagram of an example of a decoder 500. The decoder 500 can be implemented in the receiving station 106, for example, by providing a computer software program stored in the memory 204. The computer software program can include machine instructions that, when executed by a processor such as the processor 202, cause the receiving station 106 to receive the received signal in the form of a decoder 500. Figure 5 The decoder 500 can also be implemented in hardware included in, for example, the sending station 102 or the receiving station 106.

[0051] Similar to the reconstruction path of the encoder 400 discussed above, the decoder 500, in one example, includes the following stages that perform various functions to produce an output video stream 516 from the compressed bitstream 420: an entropy decoding stage 502, a dequantization stage 504, an inverse transform stage 506, an intra / inter prediction stage 508, a reconstruction stage 510, a loop filtering stage 512, and a post filter stage 514. Other structural variations of the decoder 500 can be used to decode the compressed bitstream 420.

[0052] When the compressed bitstream 420 is presented for decoding, the data elements within the compressed bitstream 420 can be decoded by an entropy decoding stage 502 to produce a set of quantized transform coefficients. A dequantization stage 504 dequantizes the quantized transform coefficients (e.g., by multiplying the quantized transform coefficients by a quantizer value), and an inverse transform stage 506 inverse transforms the dequantized transform coefficients to produce a derivative residual, which can be the same as the derivative residual created by the inverse transform stage 412 in the encoder 400. Using the header information decoded from the compressed bitstream 420, the decoder 500 can use an intra / inter prediction stage 508 to create the same prediction block as was created in the encoder 400 (e.g., at the intra / inter prediction stage 402).

[0053] At reconstruction stage 510, the predicted block can be added to the derivative residual to create a reconstructed block. A loop filter stage 512 can be applied to the reconstructed block to reduce block artifacts. Other filtering can be applied to the reconstructed block. In this example, a post filter stage 514 is applied to the reconstructed block to reduce block distortion, and the result is output as an output video stream 516. The output video stream 516 can also be referred to as a decoded video stream, and these terms will be used interchangeably herein.

[0054] Other variations of decoder 500 may be used to decode compressed bitstream 420. In some embodiments, decoder 500 may produce output video stream 516 without post filter stage 514. In some embodiments, decoder 500 may have separate learning and filtering paths. Figure 7 Embodiments and examples of a decoder having separate learning and filtering paths are shown and described.

[0055] refer to Figure 4 and Figure 5 , some stages of the encoder 400 and decoder 500 perform injective functions, while other stages of the encoder 400 and decoder 500 (e.g., loop filter stage 416 and loop filter stage 512) perform non-injective functions. With an injective function, the desired output is based on the input, so that two inputs will not give the same output. However, with a non-injective function, it is possible for two different inputs to give the same output.

[0056] Loop filter stage 416 and loop filter stage 512 perform non-injective functions designed to upscale or smooth video frames, such as to make those video frames look better to the human eye. However, this upscaling and smoothing is detrimental to the learning model because the learning model is typically a classifier. The learning model attempts to classify the input into one of several categories. Therefore, blurring the input can cause the learning model to classify inaccurately. Referring to the following data processing inequality:

[0057] I(f(X;Y)≤I(X;Y)

[0058] Where X and Y are random variables, f(·) is an arbitrary deterministic function, I(X; Y) represents the mutual information between X and Y, and I(f(X); Y) represents the mutual information between f(X) and Y. When f(·) is injective, the data processing inequality becomes the following data processing equation:

[0059] I(f(X);Y)=I(X;Y)

[0060] Furthermore, the data processing inequality becomes the following data processing strict inequality:

[0061] I(f(X;Y)<I(X;Y)

[0062] When there are two symbols x1≠x2 such that

[0063] and f(x1)=f(x2)

[0064] in, is a random variable obtained from X as follows:

[0065]

[0066] In the context of machine learning, variable X represents the input to the machine learning aspect, and variable Y represents the output of the machine learning aspect. We can process X and Y to determine how related they are. The more mutual information there is between X and Y, the more related they are. The higher the correlation between X and Y, the more likely machine learning will perform better. Therefore, the correlation represents a measure of how well machine learning will perform.

[0067] Because non-injective functions have a one-to-many mapping, the amount of mutual information between input and output is not very high, so the correlation between X and Y is necessarily low. Therefore, both the above data processing inequality and data processing equality suggest that injective data processing is allowed, but the main purpose may be to reduce the computational complexity in training and / or inference. Similarly, the above data processing inequality and data processing strict inequality suggest that non-injective data processing may have a negative impact on learning performance, for example, in terms of classification accuracy and / or learning complexity.

[0068] Reference now Figure 6 and Figure 7 , describes implementations and examples of encoders and decoders with separate learning and filtering paths. Figure 6 6 is a block diagram of an example of an encoder 600 including separate learning and filtering paths. The encoder 600 includes an intra / inter prediction stage 602, a transform stage 604, a quantization stage 606, and an entropy coding stage 608, which may be, for example, Figure 4 The intra / inter prediction stage 402, transform stage 404, quantization stage 406 and entropy coding stage 408 are shown. The encoder 600 receives Figure 3 An input video stream 300 is shown and is processed using an intra / inter prediction stage 602, a transform stage 604, a quantization stage 606, and an entropy coding stage 608 to produce a compressed bitstream 610.

[0069] The encoder 600 further includes a reconstruction path as shown by the dashed connecting line. The reconstruction path includes a dequantization stage 612, an inverse transform stage 614 and a reconstruction stage 616, which can be, for example, Figure 4The dequantization stage 410, the inverse transform stage 412, and the reconstruction stage 414 are shown. The reconstruction path of the encoder 600 includes two post-reconstruction passes for processing the output of the reconstruction stage 616. The first post-reconstruction pass (also referred to as the first pass) includes a loop filter stage 618, which can be, for example, Figure 4 Loop filter stage 416 is shown.

[0070] The second post-reconstruction path (also referred to as the second path) includes a learning model stage 620. The learning model stage 620 trains a learning model using the output of the reconstruction stage 616. Alternatively or in addition, the learning model stage 620 uses the learning model to perform reasoning operations on the output of the reconstruction stage 616. For example, the learning model stage 620 can train the learning model to identify video content and / or image content within a reconstructed frame produced using the reconstruction stage 616. The learning model refers to a machine learning model or other machine learning or artificial intelligence method. For example, the learning model can be or include a neural network, a deep neural network, a support vector machine, a cluster network, a Bayesian network, or another method.

[0071] In some cases, information associated with the loop filtering stage 618 may be used at the learning model stage 620. For example, the learning model stage 620 may use filter side information from the loop filtering stage 618. The filter side information indicates parameters, configurations, or other information used to filter the reconstructed frame at the loop filtering stage 618. In this case, the operations performed at the learning model stage 620 may include determining whether to use the filter side information for learning. If it is determined that the filter side information is not to be used for learning, the filter side information may be discarded or ignored.

[0072] The output of the first post-reconstruction pass is fed back to the intra / inter prediction stage 602. Therefore, the output of the first post-reconstruction pass can be used to generate the compressed bitstream 610. In contrast, the output of the second post-reconstruction pass is output as a learning output 622. The learning output 622 is not used to generate the compressed bitstream 610. More specifically, the learning output 622 represents the output of the learning model stage 620. The output of the learning model stage 620 can refer to the output from the training operation performed at the learning model stage 620, the output from the inference operation performed at the learning model stage 620, or both.

[0073] Compared to training the learning model using the filtered frames generated in the loop filtering stage 618, training the learning model using the reconstructed frames (e.g., without filtering them) at the learning model stage 620 results in a larger upper limit on the classification accuracy of the learning model. That is, filtering the reconstructed frames may cause some information of the reconstructed frames to be lost (e.g., by reducing the blockiness therein, etc.). Therefore, the amount of mutual information between the reconstructed frames and the filtered frames may be reduced. The amount of mutual information between the reconstructed frames used as input to the learning model stage 620 and the learning output 622 may be higher than the amount of mutual information between the filtered frames and the learning output 622. Therefore, the correlation between the reconstructed frames and the learning output 622 is higher than the correlation between the filtered frames and the learning output 622.

[0074] In some embodiments, the second post-reconstruction pass may start at the input of the loop filter stage 618. For example, instead of the second post-reconstruction pass starting as a separate output of the reconstruction stage 616, the second post-reconstruction pass and the first post-reconstruction pass may both start as the same output of the reconstruction stage 616. Thus, the input of the loop filter stage 618 may also be used to start the second post-reconstruction pass.

[0075] In some embodiments, the loop filter stage 618 may be replaced by another filter stage. For example, the loop filter stage 618 may be replaced by a deblocking filter stage, a denoising filter stage, a sample adaptive offset filter stage, a post filter stage, another filter stage, or a combination thereof.

[0076] Figure 7 700 includes a separate learning path and a filtering path. The decoder 700 receives a compressed bit stream 702, which may be, for example, a compressed bit stream using Figure 6 The encoder 600 shown produces a compressed bitstream 610. The decoder 700 uses an entropy decoding stage 704, a dequantization stage 706, an inverse transform stage 708, a reconstruction stage 710, and an intra / inter prediction stage 712 (for example, these stages can be Figure 5 The entropy decoding stage 502, dequantization stage 504, inverse transform stage 506, reconstruction stage 510 and intra / inter prediction stage 508) are shown to decode encoded data from the compressed bitstream 702.

[0077] The decoder 700 further comprises two post-reconstruction paths for processing the output of the reconstruction stage 710. The first post-reconstruction path (also referred to as the first path) of the decoder 700 comprises a loop filter stage 714 and a post-filter stage 716, which stages may be, for example, Figure 5The loop filter stage 512 and the post filter stage 514 are shown. The output of the first post reconstruction path of the decoder 700 is an output video stream 718. The output video stream 718 is a video stream that is output for display on a user device. In some cases, the post filter stage 716 can be omitted from the first post reconstruction path of the decoder 700.

[0078] The second post-reconstruction pass (also referred to as the second pass) of the decoder 700 includes a learning model stage 720. For example, the learning model stage 720 may perform the same Figure 6 6. The learning model stage 720 of the encoder 600 shown in FIG. 6 is used to perform the same or similar operations as those performed by the learning model stage 620 of the encoder 600. Thus, the learning model stage 720 uses the output of the reconstruction stage 710 to train the learning model. Alternatively or in addition, the learning model stage 720 uses the learning model to perform inference operations on the output of the reconstruction stage. The output of the learning model stage 720 is the learning output 722. The learning output 722 can be used to perform inference operations on the output of the reconstruction stage. Figure 6 The learning model stages 620 shown produce the same or similar learning outputs 622 .

[0079] In some cases, information associated with the loop filtering stage 714 can be used at the learning model stage 720. For example, the learning model stage 720 can use filter side information from the loop filtering stage 714. The filter side information indicates parameters, configurations, or other information used to filter the reconstructed frame at the loop filtering stage 714. In this case, the operations performed at the learning model stage 720 may include determining whether to use the filter side information for learning. If it is determined that the filter side information is not used for learning, the filter side information may be discarded or ignored. The learning model stage 720 may receive the filter side information from the loop filtering stage 714. Alternatively, the learning model stage 720 may receive the filter side information from the entropy decoding stage 704.

[0080] Similar to what was described above with respect to the learning output 622, training the learning model at the learning model stage 720 using the second copy of the reconstructed frame may result in a greater upper limit on the classification accuracy of the learning model than training the learning model using the filtered frame produced at the loop filtering stage 714. This is because the amount of mutual information between the reconstructed frame used as input to the learning model stage 720 and the learning output 722 may therefore be higher than the amount of mutual information between the filtered frame and the learning output 722.

[0081] In some embodiments, the second post-reconstruction pass may start at the input of the loop filter stage 714. For example, instead of the second post-reconstruction pass starting as a separate output of the reconstruction stage 710, the second post-reconstruction pass and the first post-reconstruction pass may both start as the same output of the reconstruction stage 710. Thus, the input of the loop filter stage 714 may also be used to start the second post-reconstruction pass.

[0082] Further details of the technique for video coding using separate learning and filtering paths are now described. Figure 8 is a flow chart of an example of a technique 800 for video coding using separate learning and filtering paths. Fig. 9 is a flow diagram of an example of a technique 900 for training and inference using a learning model for video coding.

[0083] For example, technique 800 and / or technique 900 may be implemented as a software program that may be executed by a computing device, such as sending station 102 or receiving station 106. For example, the software program may include machine-readable instructions that may be stored in a memory, such as memory 204 or auxiliary storage 214, and when executed by a processor, such as processor 202, may cause a computing device to perform technique 800 and / or technique 900. Technique 800 and / or technique 900 may be implemented using dedicated hardware or firmware. For example, hardware components are configured to perform technique 800 and / or technique 900. As described above, some computing devices may have multiple memories or processors, and the operations described in technique 800 and / or technique 900 may be distributed using multiple processors, memories, or both.

[0084] For ease of explanation, both technology 800 and technology 900 are depicted and described as a series of steps or operations. However, the steps or operations according to the present disclosure can occur in various orders and / or simultaneously. In addition, other steps or operations not presented and described herein can be used. In addition, all the steps or operations shown may not be required to implement the technology according to the disclosed subject matter.

[0085] First reference Figure 8 , a technique 800 for video coding using separate learning and filtering paths is shown. At 802, quantized transform coefficients representing video and / or image data are dequantized. For example, the quantized transform coefficients may represent quantized transform coefficients of a video block to be encoded or an encoded video block to be decoded. The transform coefficients are generated as a result of the dequantization. At 804, the transform coefficients are inversely transformed. Prediction residuals are generated as a result of the inverse transform. At 806, the prediction residuals are reconstructed to produce a reconstructed frame. The reconstructed frame is a video frame or image that includes a video block or image block reconstructed based on the prediction residuals.

[0086] There are two paths starting at 806. The first path includes the operations performed in 808 and 810. The second path includes the operations performed in 812 and 814. Both the first path and the second path perform their respective operations on a reconstructed frame (e.g., the reconstructed frame generated in 806). Therefore, the functionality of the first path can be described with respect to a first copy of the reconstructed frame, and the functionality of the second path can be described with respect to a second copy of the reconstructed frame. However, the use of the term "copy" in this context does not necessarily mean that the reconstructed frame (e.g., the reconstructed frame generated in 806) has actually been copied. Rather, the terms "first copy" and "second copy" are used only to indicate that both the first path and the second path can receive the same reconstructed frame as input.

[0087] According to the first path, in 808, the first copy of the reconstructed frame is filtered. Filtering the reconstructed frame includes using one or more filtering techniques. The filtering technique refers to a non-injective function performed using one or more filters of a video or image codec. For example, the filter can be an in-loop filter, a deblocking filter, a denoising filter, a sample adaptive offset filter, a post filter, another filter, or a combination thereof. The filtered frame is generated by filtering the first copy of the reconstructed frame. In 810, the filtered frame is output. For example, the filtered frame can be included in an output video stream for display on a user device.

[0088] According to the second path, in 812, the second copy of the reconstructed frame is processed using the learning model. Processing the second copy of the reconstructed frame using the learning model may include training the learning model using the second copy of the reconstructed frame. For example, training the learning model using the second copy of the reconstructed frame includes training the learning model using the second copy of the reconstructed frame to identify video content. The learning model classifies the content of the second copy of the reconstructed frame. The output of this classification can then be used to identify video content, such as objects or other visual effects in other video frames.

[0089] Additionally or alternatively, processing the second copy of the reconstructed frame using the learning model may include performing one or more reasoning operations on the second copy of the reconstructed frame using the trained learning model. For example, performing reasoning operations on the second copy of the reconstructed frame using the trained learning model may include classifying the second copy of the reconstructed frame according to the trained classification model, determining a class index in the classification performed thereon, or identifying video semantics associated with the second copy of the reconstructed frame using a trained video semantics model.

[0090] At 814, the learning model data is output. For example, the output learning model data may refer to data associated with training the learning model, such as residual training error, accuracy of classification on a validation set, or both. In another example, the output learning model data may refer to data associated with inference performed using the trained learning model, such as class indices in classification, labeled segments or annotations associated with some or all of the second copy of the reconstructed frame, or both.

[0091] Each function performed at the dequantization stage, the inverse transform stage, the reconstruction stage, and the second post-reconstruction pass is an injective function. However, some or all of the functions performed in the first pass (e.g., the filter function used to generate the filtered frame) are non-injective functions. Therefore, the filtered frame is generated using at least one non-injective function, and the second copy of the reconstructed frame is generated without using the non-injective function. Therefore, training the learning model using the second copy of the reconstructed frame results in a larger upper limit on the classification accuracy of the learning model than training the learning model using the filtered frame.

[0092] In some embodiments, the operation of training the learning model may include training the learning model using the second copy of the reconstructed frame and the filter side information. For example, the filter side information may indicate information about the filtering performed in the first pass. The filter side information may be received from the first pass (e.g., from an in-loop filter or other filter that performs filtering on the first pass). Alternatively, the filter side information may also be information for filtering the first copy of the reconstructed frame in the first pass.

[0093] In some such embodiments, the operation of training the learning model using the second copy of the reconstructed frame and the filter side information may include determining whether the filter side information is useful for training the learning model. If it is determined that the filter side information is not useful for training the learning model, the filter side information may be discarded or ignored.

[0094] In some embodiments, the second path may include training the learning model. For example, training the learning model may include comparing the output of the learning model with the expected output. That is, a cost function may be used to measure the difference between the actual learning model output and the expected learning model output. The value generated by the cost function may then be used to train the learning model.

[0095] In some embodiments, technique 800 may include an operation of entropy decoding syntax elements from an encoded bitstream before dequantizing coefficients at 802. For example, in the case where technique 800 is performed using a decoder ( Figure 7In the case where the technique 800 is performed by a decoder 700 shown in the drawings or is used to decode encoded video and / or image data (such as encoded video frames), the technique 800 can begin by entropy decoding syntax elements representing encoded video frames from an encoded bitstream (such as to produce quantized transform coefficients). The quantized transform coefficients can then be dequantized. Each function performed at the entropy decoding stage is an injective function.

[0096] In some embodiments, the second path may include performing an inference operation on the second copy of the reconstructed frame or other encoded video data or encoded image data. For example, instead of training the learning model using the second copy of the reconstructed frame, the learning model may have been trained. Therefore, the learning model can be used to reason about the second copy of the reconstructed frame, such as to identify its content. In some such embodiments, the second path may omit the operation of training the learning model. Therefore, the output of the second path will be inference data, rather than training data.

[0097] Next reference Fig. 9 , shows a technique 900 for inference using a learning model for video coding. Specifically, the technique 900 describes a training operation and an inference operation, both of which are performed in an encoder or decoder (e.g. Figure 6 The encoder 600 or Figure 7 For example, the training operation and the inference operation described with respect to the technique 900 may be performed at different times, such as where the training operation is performed before the inference operation.

[0098] At 902, a first reconstructed frame is generated based on first encoded video data. Generating the first reconstructed frame may include dequantizing quantized transform coefficients to generate transform coefficients, inverse transforming the transform coefficients to generate prediction residuals, and reconstructing the prediction residuals to generate the first reconstructed frame. At 904, the first reconstructed frame is used to train a learning model. In some cases, the learning model may also be trained using filter side information.

[0099] At 906, a second reconstructed frame is generated based on the second encoded video data. Generating the second reconstructed frame may include dequantizing the quantized transform coefficients to generate transform coefficients, inverse transforming the transform coefficients to generate prediction residuals, and reconstructing the prediction residuals to generate the second reconstructed frame. At 908, one or more inference operations are performed on the second reconstructed frame using the trained learning model.

[0100] The first coded video data and the second coded video data may or may not be derived from a common source. For example, the first coded video data may correspond to a first video frame of a video stream or bitstream, and the second coded video data may correspond to a second video frame of the same video stream or bitstream. In the coding and / or display order of the video stream or bitstream, the second video frame is later than the first video frame (e.g., immediately after or after the first video frame). In another example, the first coded video data may correspond to a video frame of a first video stream or bitstream, and the second coded video data may correspond to a video frame of a second video stream or bitstream. The second video stream or second bitstream may be processed at a time after the first video stream or the second bitstream.

[0101] The various aspects of encoding and decoding described above illustrate some examples of encoding and decoding techniques. However, it is to be understood that when those terms are used in the claims, encoding and decoding can mean compression, decompression, transformation or any other processing or change of data.

[0102] The word "example" is used herein to mean used as an example, instance or illustration. Any aspect or design described as an "example" in this article is not necessarily interpreted as being superior to other aspects or designs as being preferred or advantageous. On the contrary, the use of the word "example" is intended to present the concept in a specific way. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clearly indicated by the context, the statement "X includes A or B" is intended to mean any one of its natural inclusive arrangements. That is, if X includes A; X includes B; or X includes both A and B, then "X includes A or B" is satisfied under any of the above examples. In addition, unless otherwise specified or indicated by the context as being directed to a singular form, the articles "one" and "an" as used in this application and the appended claims should generally be interpreted as meaning "one or more". In addition, the use of the term "embodiment" or the term "one embodiment" throughout this disclosure is not intended to mean the same embodiment unless described as such.

[0103] The implementation of the sending station 102 and / or the receiving station 106 (and the algorithms, methods, instructions, etc. stored thereon and / or executed thereby (including by the encoder 400 or 600 and the decoder 500 or 700)) can be implemented with hardware, software or any combination thereof. The hardware may include, for example, a computer, an intellectual property (IP) core, an application specific integrated circuit (ASIC), a programmable logic array, an optical processor, a programmable logic controller, a microcode, a microcontroller, a server, a microprocessor, a digital signal processor, or any other suitable circuit. In the claims, the term "processor" should be understood to include any one of the above-mentioned hardware alone or in combination. The terms "signal" and "data" are used interchangeably. In addition, the parts of the sending station 102 and the receiving station 106 do not necessarily have to be implemented in the same manner.

[0104] In addition, in one aspect, for example, a general purpose computer or general purpose processor with a computer program can be used to implement the sending station 102 or the receiving station 106, which, when executed, performs any of the various methods, algorithms and / or instructions described herein. In addition or alternatively, for example, a special purpose computer / processor can be utilized, which can include other hardware for performing any of the methods, algorithms or instructions described herein.

[0105] Transmitting station 102 and receiving station 106 can be realized on the computer in the video conference system, for example.Alternatively, transmitting station 102 can be realized on a server, and receiving station 106 can be realized on a device (such as a handheld communication device) separated from the server.In this example, transmitting station 102 can use encoder 400 or 600 to encode content into a coded video signal and send the coded video signal to a communication device.And then, the communication device can use decoder 500 or 700 to decode the coded video signal.Alternatively, the communication device can decode the content (for example, not sent by transmitting station 102) stored locally on the communication device.Other suitable transmission and reception implementation schemes are available.For example, receiving station 106 can be a usually fixed personal computer rather than a portable communication device, and / or the equipment comprising encoder 400 or 600 can also include decoder 500 or 700.

[0106] In addition, all or part of the implementation of the present disclosure may take the form of a computer program product accessible, for example, from a computer usable medium or a computer readable medium. A computer usable medium or a computer readable medium may be any device that may, for example, tangibly contain, store, transmit, or convey a program for use by or in conjunction with any processor. The medium may be, for example, an electronic, magnetic, optical, electromagnetic, or semiconductor device. Other suitable media are also available.

[0107] The above embodiments and other aspects have been described to facilitate easy understanding of the present disclosure and do not limit the present disclosure. On the contrary, the present disclosure is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which scope should be given the broadest interpretation permitted under the law so as to include all such modifications and equivalent arrangements.

Claims

1. A decoder, comprising: an entropy decoding stage configured to entropy decode syntax elements representing an encoded video frame from an encoded bitstream to produce quantized transform coefficients; a dequantization stage configured to dequantize the quantized transform coefficients to produce transform coefficients; an inverse transform stage configured to inversely transform the transform coefficients to produce a prediction residual; a reconstruction stage configured to reconstruct the prediction residual to produce a reconstructed frame; a first post-reconstruction path configured to filter the first copy of the reconstructed frame using one or more filtering techniques and output a filtered frame within an output video stream for display on a user device; as well as and a second post-reconstruction pathway configured to train a learning model for video content identification using a second copy of the reconstructed frame and output a trained learning model, wherein training the learning model using the second copy of the reconstructed frame results in a greater upper classification accuracy limit for the learning model compared to training the learning model using the filtered frame.

2. The decoder according to claim 1, wherein: The second post-reconstruction path is configured to train the learning model using the second copy of the reconstructed frame and filter-side information received from the first post-reconstruction path.

3. The decoder according to claim 1, wherein: Each function performed at the entropy decoding stage, the dequantization stage, the inverse transform stage, the reconstruction stage and the second post-reconstruction pass is an injective function, wherein at least one function performed at the first post-reconstruction pass is a non-injective function.

4. The decoder according to claim 1, wherein: The reconstructed frame is a first reconstructed frame, wherein the reconstruction stage is further configured to produce a second reconstructed frame, wherein the first post-reconstruction path is configured to filter a first copy of the second reconstructed frame, and wherein the second post-reconstruction path is configured to perform one or more inference operations on a second copy of the second reconstructed frame using a trained learning model.

5. The decoder according to claim 1, wherein: The one or more filtering techniques include filtering techniques performed using an in-loop filter.

6. A method comprising: dequantizing quantized transform coefficients representing encoded video data to produce transform coefficients; performing an inverse transformation on the transform coefficients to generate a prediction residual; reconstructing the prediction residual to generate a reconstructed frame; filtering the first copy of the reconstructed frame to produce a filtered frame for inclusion in an output video stream; as well as A learning model is trained and outputted, wherein training the learning model includes processing the second copy of the reconstructed frame to identify video content.

7. The method according to claim 6, wherein: Training and outputting the learning model includes: The learning model is trained using the second copy of the reconstructed frame to identify the video content.

8. The method of claim 7, wherein training the learning model using the second copy of the reconstructed frame results in a greater upper classification accuracy limit for the learning model than training the learning model using the filtered frame.

9. The method according to claim 7, wherein: Training the learning model using the second copy of the reconstructed frame to identify the video content includes: The learning model is trained using filter side information, wherein the filter side information is used to filter the first copy of the reconstructed frame.

10. The method according to claim 6, wherein: The filtered frame is generated using at least one non-injective function, wherein the second copy of the reconstructed frame is generated without using the at least one non-injective function.

11. The method according to claim 6, wherein: Filtering the first copy of the reconstructed frame to produce the filtered frame for inclusion in an output video stream comprises: The first copy of the reconstructed frame is processed using an in-loop filter.

12. The method according to claim 6, wherein: The encoded video data is first encoded video data, and the method further comprises: One or more inference operations are performed on the second encoded video data using the trained learning model.

13. The method according to claim 6, wherein: The coded video data is decoded from a coded bit stream, the method further comprising: The filtered frames are output in the output video stream for display on a user device.

14. An integrated circuit comprising a processor for executing instructions, the instructions comprising instructions for: decoding the encoded video data from the encoded bitstream to produce a reconstructed frame; processing a first copy of the reconstructed frame via a first decoding pass using an in-loop filter to produce an output video stream for display on a user device; and A second copy of the reconstructed frame is processed via a second decoding pass to train and output a learned model for identifying video content.

15. The integrated circuit of claim 14, wherein: The instructions for decoding the encoded video data from the encoded bitstream to produce the reconstructed frame include instructions for: dequantizing quantized transform coefficients representing the encoded video data to produce transform coefficients; performing an inverse transformation on the transform coefficients to generate a prediction residual; as well as The prediction residual is reconstructed to generate a reconstructed frame.

16. The integrated circuit of claim 14, wherein: The instructions for processing the second copy of the reconstructed frame via the second decoding path to train and output a learned model for identifying the video content include instructions for: The learning model is trained using the second copy of the reconstructed frame.

17. The integrated circuit of claim 16, wherein: The instructions for training the learning model using the second copy of the reconstructed frame include instructions for: The learning model is trained using filter-side information from the first decoding path.

18. The integrated circuit of claim 14, wherein: The encoded video data is decoded from an encoded bitstream, and the instructions include instructions for: One or more inference operations are performed on the second encoded video data using the trained learning model.

19. The integrated circuit of claim 14, wherein: Training the learning model by processing the second copy of the reconstructed frame results in a greater upper limit of classification accuracy of the learning model compared to training the learning model by processing a filtered frame, wherein the filtered frame is generated by filtering the first copy of the reconstructed frame using the in-loop filter.

20. The integrated circuit of claim 14, wherein: Processing the first copy of the reconstructed frame via the first decoding pass includes using at least one non-injective function, wherein processing the second copy of the reconstructed frame via the second decoding pass omits using the at least one non-injective function.

Images