Method, device and storage medium for encoding input video stream

Abstract

Adaptive filtering is used for video streams to reduce bit rate. A first copy of an input video stream is encoded as a reference bit stream. Each of a plurality of candidate filters is applied to each frame of a second copy of the input video stream to generate a filtered second copy of the input video stream. The filtered second copy is encoded as a candidate bit stream. The cost value of the candidate filter is determined based on the distortion value and the bit rate difference between the candidate bit stream and the reference bit stream. The candidate bit stream corresponding to the candidate filter with the lowest cost value among the cost values ​​is selected as the output bit stream, which is then output or stored. Using an adaptive filter and processing the input video stream before encoding may result in a reduction in bit rate, thereby improving compression, decompression and other performances.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims the benefit of U.S. Provisional Application No. 62 / 784,626, filed on December 24, 2018, the entire disclosure of which is incorporated herein by reference. Background Art

[0003] A digital video stream can represent a video using a sequence of frames or still images. Digital video can be used for a variety of applications, including, for example, video conferencing, high-definition video entertainment, video advertising, or user-generated video sharing. A digital video stream can contain a large amount of data and consume a large amount of computing or communication resources of a computing device to process, transmit, or store video data. Various methods have been proposed to reduce the amount of data in a video stream, including encoding or decoding techniques. Summary of the invention

[0004] According to an embodiment of the present disclosure, a method for encoding an input video stream into an output bitstream includes encoding a first copy of the input video stream as a reference bitstream. The method further includes comparing the input video stream with the reference bitstream to determine a first distortion value and a first bitrate of the reference bitstream. The method further includes, for each candidate filter among a plurality of candidate filters: applying the candidate filter to each frame of the second copy of the input video stream to generate a filtered second copy of the input video stream; encoding the filtered second copy of the input video stream as a candidate bitstream; comparing the input video stream with the candidate bitstream to determine a second distortion value and a second bitrate of the candidate bitstream; and determining a cost value of the candidate filter based on a difference between the first distortion value and the second distortion value and based on a difference between the first bitrate and the second bitrate. The method further includes selecting a candidate bitstream corresponding to the candidate filter having the lowest cost value among the cost values ​​as the output bitstream. The method further includes outputting or storing the output bitstream.

[0005] In some embodiments of the method, the candidate filters are obtained using at least one of a genetic algorithm, a gradient descent algorithm, or another nonlinear optimization technique.

[0006] In some embodiments of the method, applying a given candidate filter to a given frame of a second copy of the input video stream to produce a filtered second copy of the input video stream includes: transforming the values ​​within the given frame from the spatial domain to the frequency domain; after the transformation, applying the given candidate filter to each block of the given frame to produce filtered frequency domain values; and inverse transforming the filtered frequency domain values ​​from the frequency domain to the spatial domain.

[0007] In some embodiments of the method, applying the given candidate filter to each block of the frame to generate the filtered frequency domain value includes denoising overlapping block regions of the frame using the given candidate filter.

[0008] In some embodiments of the method, the overlapping block regions are semi-overlapping block regions.

[0009] In some embodiments of the method, determining a cost value for a given candidate filter among the multiple candidate filters includes: calculating a weighted value by applying a weight to the difference between the first distortion value and the second distortion value; and determining the cost value by subtracting the weighted value from the difference between the first bit rate and the second bit rate.

[0010] In some embodiments of the method, the weight is defined based on a target for the second distortion value.

[0011] According to an embodiment of the present disclosure, a device for encoding an input video stream into an output bitstream includes a memory and a processor, which is configured to execute instructions stored in the memory. The instruction includes an instruction to encode a first copy of the input video stream into a reference bitstream. The instruction further includes an instruction to compare the input video stream with the reference bitstream to determine a first distortion value and a first bitrate of the reference bitstream. The instruction further includes instructions for the following: for each candidate filter in a plurality of candidate filters: applying the candidate filter to each frame of the second copy of the input video stream to generate a filtered second copy of the input video stream; encoding the filtered second copy of the input video stream into a candidate bitstream; comparing the input video stream with the candidate bitstream to determine a second distortion value and a second bitrate of the candidate bitstream; and determining a cost value of the candidate filter based on the difference between the first distortion value and the second distortion value and based on the difference between the first bitrate and the second bitrate. The instruction further includes an instruction to select a candidate bitstream corresponding to the candidate filter having the lowest cost value among the cost values ​​as the output bitstream. The instruction further includes an instruction to output or store the output bitstream.

[0012] In some embodiments of the apparatus, the candidate filters are obtained using at least one of a genetic algorithm, a gradient descent algorithm, or another nonlinear optimization technique.

[0013] In some embodiments of the device, instructions for applying a given candidate filter to a given frame of a second copy of the input video stream to produce a filtered second copy of the input video stream include instructions for performing the following operations: transforming values ​​within the given frame from the spatial domain to the frequency domain; after the transformation, applying the given candidate filter to each block of the given frame to produce filtered frequency domain values; and inversely transforming the filtered frequency domain values ​​from the frequency domain to the spatial domain.

[0014] In some embodiments of the apparatus, the instructions for applying the given candidate filter to each block of the frame to generate the filtered frequency domain values ​​include instructions for performing the following operations: denoising overlapping block regions of the frame using the given candidate filter.

[0015] In some embodiments of the device, the overlapping block regions are semi-overlapping block regions.

[0016] In some embodiments of the device, the instructions for determining a cost value for a given candidate filter among the multiple candidate filters include instructions for performing the following operations: calculating a weighted value by applying a weight to the difference between the first distortion value and the second distortion value; and determining the cost value by subtracting the weighted value from the difference between the first bit rate and the second bit rate.

[0017] In some embodiments of the apparatus, the weight is defined based on a target for the second distortion value.

[0018] According to an embodiment of the present disclosure, an encoder for encoding an input video stream into an output bitstream performs an operation including encoding a first copy of the input video stream into a reference bitstream. The operation further includes comparing the input video stream with the reference bitstream to determine a first distortion value and a first bitrate of the reference bitstream. The operation further includes, for each candidate filter among a plurality of candidate filters: applying the candidate filter to each frame of a second copy of the input video stream to generate a filtered second copy of the input video stream; encoding the filtered second copy of the input video stream into a candidate bitstream; comparing the input video stream with the candidate bitstream to determine a second distortion value and a second bitrate of the candidate bitstream; and determining a cost value of the candidate filter based on a difference between the first distortion value and the second distortion value and based on a difference between the first bitrate and the second bitrate. The operation further includes selecting a candidate bitstream corresponding to the candidate filter having the lowest cost value among the cost values ​​as the output bitstream. The operation further includes outputting or storing the output bitstream.

[0019] In some embodiments of the encoder, the candidate filters are obtained using at least one of a genetic algorithm, a gradient descent algorithm, or another nonlinear optimization technique.

[0020] In some embodiments of the encoder, applying a given candidate filter to a given frame of a second copy of the input video stream to produce a filtered second copy of the input video stream includes: transforming the values ​​within the given frame from the spatial domain to the frequency domain; after the transformation, applying the given candidate filter to each block of the given frame to produce filtered frequency domain values; and inverse transforming the filtered frequency domain values ​​from the frequency domain to the spatial domain.

[0021] In some embodiments of the encoder, applying the given candidate filter to each block of the frame to generate the filtered frequency domain value comprises denoising overlapping block regions of the frame using the given candidate filter.

[0022] In some embodiments of the encoder, the overlapping block regions are half-overlapping block regions.

[0023] In some embodiments of the encoder, determining a cost value for a given candidate filter among the multiple candidate filters includes: calculating a weighted value by applying a weight to the difference between the first distortion value and the second distortion value; and determining the cost value by subtracting the weighted value from the difference between the first bit rate and the second bit rate.

[0024] In some embodiments of the encoder, the weight is defined based on a target for the second distortion value.

[0025] These and other aspects of the disclosure are disclosed in the following detailed description of embodiments, the appended claims and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The description herein refers to the drawings described below, wherein like reference numerals refer to like parts throughout the several views.

[0027] Figure 1 is a schematic diagram of an example of a video encoding and decoding system.

[0028] Figure 2 is a block diagram of an example of a computing device that may implement a transmitting station or a receiving station.

[0029] Figure 3 is a diagram of an example of a video stream to be encoded and then decoded.

[0030] Figure 4 is a block diagram of an example of an encoder according to an embodiment of the present disclosure.

[0031] Figure 5is a block diagram of an example of a decoder according to an embodiment of the present disclosure.

[0032] Figure 6 is a block diagram of an example of a video stream processing pipeline that uses adaptive filtering to reduce bit rate.

[0033] Figure 7 is a block diagram of an example of operations performed to select an output bitstream using adaptive filtering.

[0034] Fig. 8A -B is a flow chart of an example of a technique for encoding a video stream into an output bitstream using adaptive filtering.

[0035] Fig. 9 is a flow diagram of an example of a technique for selecting an output bitstream based on cost values ​​determined for candidate filters. DETAILED DESCRIPTION

[0036] Video compression schemes may include breaking down a corresponding image or frame into smaller parts such as blocks, and generating a compressed bitstream using techniques that limit the information included in the output for the corresponding blocks. The compressed bitstream may be decoded to recreate the source image from the limited information. Typical video compression techniques include reducing the bitrate of a video stream, such as to reduce transmission costs for video hosting and service platforms.

[0037] However, reducing the bitrate of a video stream often results in a loss of quality, making the quality of the subsequently decompressed video stream lower than the quality of the original video stream processed using the compression technique. For example, undesirable blocking artifacts may be introduced during the compression of the video stream. In some cases, those blocking artifacts may remain in the video stream after it is decompressed.

[0038] In some cases, pre-processing can be performed on the video stream before the video stream is compressed using a video compression scheme. Performing pre-processing can result in improving the video quality of a fixed bit rate video stream, or minimizing the bit rate of a fixed quality video stream. For example, pre-processing can include applying a filter to the video stream to remove noise and reduce artifacts (e.g., blocks, strips, ringing, time flicker, etc.) in the video stream before the video stream is encoded as an output bit stream. However, this filter-based method uses a fixed filter, which depends on the selected bit rate and / or selected quality of the video stream. Therefore, typical fixed filters as used in video stream pre-processing cannot minimize bit rate or maximize quality.

[0039] Furthermore, the parameters used by such fixed filters depend on the content of the image blocks within the frames of the video stream. Thus, typical fixed filters are generally more useful for removing artifacts from video edge textures than for removing artifacts from homogeneous regions of image blocks. However, conventional methods for estimating fixed filter parameters often induce undesirable distortions within the frames of the video stream and may also affect the compression process in complex and nonlinear ways.

[0040] Embodiments of the present invention solve problems such as these by preprocessing a video stream using adaptive filtering to reduce the bit rate. A first copy of an input video stream is encoded as a reference bit stream. Each of a plurality of candidate filters is applied to each frame of a second copy of the input video stream to generate a filtered second copy of the input video stream. The filtered second copy is encoded as a candidate bit stream. The cost value of the candidate filter is determined based on the distortion value and the bit rate difference between the candidate bit stream and the reference bit stream. The candidate bit stream corresponding to the candidate filter having the lowest cost value among the cost values ​​is selected as the output bit stream, and then the bit stream is output or stored. Using an adaptive filter and processing the input video stream before encoding may result in a reduction in bit rate, thereby improving compression, decompression and other performance.

[0041] Further details of techniques for adaptive filtering of video streams to reduce bitrate are described herein first with reference to systems in which they may be implemented. Figure 1 is a schematic diagram of an example of a video encoding and decoding system 100. The transmitting station 102 may be, for example, Figure 2 , and a computer having an internal configuration of the hardware as described in . However, other implementations of the sending station 102 are possible. For example, the processing of the sending station 102 can be distributed among multiple devices.

[0042] The network 104 can connect the sending station 102 and the receiving station 106 to encode and decode the video stream. Specifically, the video stream can be encoded at the sending station 102, and the encoded video stream can be decoded at the receiving station 106. The network 104 can be, for example, the Internet. The network 104 can also be a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), a cellular telephone network, or any other means of transmitting the video stream from the sending station 102 to the receiving station 106 in this example.

[0043] In one example, receiving station 106 may be a Figure 2 , and a computer having an internal configuration of the hardware as described in . However, other suitable implementations of the receiving station 106 are possible. For example, the processing of the receiving station 106 can be distributed among multiple devices.

[0044] Other embodiments of the video encoding and decoding system 100 are possible. For example, one embodiment may omit the network 104. In another embodiment, the video stream may be encoded and then stored for transmission to a receiving station 106 or any other device with a memory at a later time. In one embodiment, the receiving station 106 receives the encoded video stream (e.g., via the network 104, a computer bus, and / or some communication path) and stores the video stream for later decoding. In an exemplary embodiment, a real-time transport protocol (RTP) is used for transmission of the encoded video over the network 104. In another embodiment, a transport protocol other than RTP may be used (e.g., a video streaming protocol based on the hypertext transfer protocol (HTTP)).

[0045] For example, when used in a video conferencing system, the sending station 102 and / or the receiving station 106 may include the ability to both encode and decode video streams as described below. For example, the receiving station 106 may be a video conference participant who receives an encoded video bitstream from a video conference server (e.g., the sending station 102) for decoding and viewing, and further encodes and sends his or her own video bitstream to the video conference server for decoding and viewing by other participants.

[0046] In some embodiments, the video encoding and decoding system 100 may alternatively be used to encode and decode data other than video data. For example, the video encoding and decoding system 100 may be used to process image data. The image data may include data blocks from an image. In such an embodiment, the sending station 102 may be used to encode the image data, and the receiving station 106 may be used to decode the image data.

[0047] Alternatively, receiving station 106 may represent a computing device that stores encoded image data for later use, such as after receiving the encoded or pre-encoded image data from sending station 102. As yet another alternative, sending station 102 may represent a computing device that decodes image data, such as before sending the decoded image data to receiving station 106 for display.

[0048] Figure 2 is a block diagram of an example of a computing device 200 that can implement a sending station or a receiving station. For example, the computing device 200 can implement Figure 1 One or both of the sending station 102 and the receiving station 106. The computing device 200 may be in the form of a computing system including multiple computing devices, or in the form of one computing device (e.g., a mobile phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, etc.).

[0049] The processor 202 in the computing device 200 may be a conventional central processing unit. Alternatively, the processor 202 may be another type of device or devices capable of manipulating or processing existing or hereafter developed information. For example, although the disclosed embodiments may be implemented using one processor (e.g., the processor 202) as shown, advantages in speed and efficiency may be achieved by using more than one processor.

[0050] In an embodiment, the memory 204 in the computing device 200 may be a read-only memory (ROM) device or a random access memory (RAM) device. However, other suitable types of storage devices may be used as the memory 204. The memory 204 may include code and data 206 accessed by the processor 202 using a bus 212. The memory 204 may further include an operating system 208 and application programs 210, which include at least one program that allows the processor 202 to perform the techniques described herein. For example, the application programs 210 may include application 1 to application N, which further include a video and / or image coding application that performs the techniques described herein.

[0051] The computing device 200 may also include auxiliary storage 214, which may be, for example, a memory card used with a mobile computing device. Because a video communication session may contain a large amount of information, all or part of it may be stored in the auxiliary storage 214 and loaded into the memory 204 as needed for processing.

[0052] The computing device 200 may also include one or more output devices, such as a display 218. In one example, the display 218 may be a touch-sensitive display that combines a display with a touch-sensitive element operable to sense touch input. The display 218 may be coupled to the processor 202 via the bus 212. In addition to or as an alternative to the display 218, other output devices that allow a user to program or otherwise use the computing device 200 may also be provided. When the output device is or includes a display, the display may be implemented in various ways, including by a liquid crystal display (LCD), a cathode ray tube (CRT) display, or a light emitting diode (LED) display, such as an organic LED (OLED) display.

[0053] The computing device 200 may also include or be in communication with an image sensing device 220, such as a camera or any other image sensing device 220 now known or hereafter developed that can sense an image, such as an image of a user operating the computing device 200. The image sensing device 220 may be positioned so that it is oriented toward the user operating the computing device 200. In an example, the position and optical axis of the image sensing device 220 may be configured so that the field of view includes an area directly adjacent to the display 218 and from which the display 218 is visible.

[0054] The computing device 200 may also include or communicate with a sound sensing device 222, such as a microphone or any other sound sensing device now known or hereafter developed that can sense sounds in the vicinity of the computing device 200. The sound sensing device 222 may be positioned so that it is oriented toward a user operating the computing device 200, and may be configured to receive sounds, such as voice or other utterances, uttered by the user while the user is operating the computing device 200.

[0055] although Figure 2 The processor 202 and memory 204 of the computing device 200 are depicted as being integrated into one unit, but other configurations may be utilized. The operations of the processor 202 may be distributed across multiple machines (where individual machines may have one or more processors) that may be coupled directly or across a local area network or other network. The memory 204 may be distributed across multiple machines (such as network-based storage or storage in multiple machines that perform the operations of the computing device 200).

[0056] Although depicted here as one bus, the bus 212 of the computing device 200 may include multiple buses. Additionally, the secondary storage 214 may be directly coupled to other components of the computing device 200 or may be accessed via a network and may include an integrated unit (such as a memory card) or multiple units (such as multiple memory cards). The computing device 200 may therefore be implemented in a wide variety of configurations.

[0057] Figure 3 is a diagram of an example of a video stream 300 to be encoded and subsequently decoded. The video stream 300 includes a video sequence 302. At the next stage, the video sequence 302 includes a plurality of adjacent frames 304. Although three frames are described as adjacent frames 304, the video sequence 302 may include any number of adjacent frames 304. The adjacent frames 304 may then be further subdivided into individual frames, such as frames 306.

[0058] At the next stage, the frame 306 may be divided into a series of planes or fragments 308. For example, the fragments 308 may be subsets of a frame, such as to allow parallel processing. The fragments 308 may also be subsets of a frame that may separate the video data into separate colors. For example, a frame 306 of color video data may include a luma plane and two chroma planes. The fragments 308 may be sampled at different resolutions.

[0059] Regardless of whether the frame 306 is divided into segments 308, the frame 306 can be further subdivided into blocks 310, which can contain data corresponding to, for example, 16x16 pixels in the frame 306. The blocks 310 can also be arranged to include data from one or more segments 308 of pixel data. The blocks 310 can also have any other suitable size, such as 4x4 pixels, 8x8 pixels, 16x8 pixels, 8x16 pixels, 16x16 pixels, or larger. Unless otherwise specified, the terms block and macroblock are used interchangeably herein.

[0060] Figure 4 4 is a block diagram of an example of encoder 400. As described above, encoder 400 may be implemented in transmission station 102, such as by providing a computer software program stored in a memory (e.g., memory 204). The computer software program may include machine instructions that, when executed by a processor such as processor 202, cause transmission station 102 to transmit the data in a manner that is consistent with the embodiment of the present invention. Figure 4 The video data is encoded in the manner described in . The encoder 400 may also be implemented as dedicated hardware included in, for example, the sending station 102. In a particularly preferred embodiment, the encoder 400 is a hardware encoder.

[0061] The encoder 400 has the following stages that perform various functions in a forward path (shown by solid connecting lines) to use the video stream 300 as input to produce an encoded or compressed bitstream 420: an intra / inter prediction stage 402, a transform stage 404, a quantization stage 406, and an entropy coding stage 408. The encoder 400 may also include a reconstruction path (shown by a dashed connecting line) to reconstruct frames for encoding future blocks. Figure 4 In FIG. 4 , the encoder 400 has the following stages that perform various functions in the reconstruction path: a dequantization stage 410, an inverse transform stage 412, a reconstruction stage 414, and a loop filter stage 416. Other structural variations of the encoder 400 may be used to encode the video stream 300.

[0062] In some cases, the functions performed by the encoder 400 may occur after filtering the video stream 300. That is, before the encoder 400 receives the video stream 300, the video stream 300 may undergo pre-processing according to one or more embodiments of the present disclosure. Alternatively, before continuing to perform the reference Figure 4 The encoder 400 itself may perform such pre-processing on the video stream 300 prior to the described functionality, such as before processing the video stream 300 at the intra / inter prediction stage 402 .

[0063] When the video stream 300 is presented for encoding after preprocessing is performed, the corresponding adjacent frames 304, such as frame 306, can be processed in units of blocks. At the intra / inter prediction stage 402, the corresponding blocks can be encoded using intra prediction (also referred to as intra prediction) or inter prediction (also referred to as inter prediction). In any case, a prediction block can be formed. In the case of intra prediction, the prediction block can be formed from samples in the current frame that have been previously encoded and reconstructed. In the case of inter prediction, the prediction block can be formed from samples in one or more previously constructed reference frames.

[0064] Next, the prediction block can be subtracted from the current block at the intra / inter prediction stage 402 to produce a residual block (also referred to as a residual). The transform stage 404 uses a block-based transform to transform the residual into transform coefficients in, for example, the frequency domain. The quantization stage 406 uses a quantizer value or quantization stage to convert the transform coefficients into discrete quantum values, which are referred to as quantized transform coefficients. For example, the transform coefficients can be divided by the quantizer value and truncated.

[0065] The quantized transform coefficients are then entropy encoded by the entropy encoding stage 408. The entropy encoded coefficients are then output to a compressed bitstream 420 along with other information used to decode the block (which may include, for example, syntax elements such as those used to indicate the prediction type used, the transform type, motion vectors, quantizer values, etc.). The compressed bitstream 420 may be formatted using various techniques such as variable length coding (VLC) or arithmetic coding. The compressed bitstream 420 may also be referred to as an encoded video stream or an encoded video bitstream, and these terms will be used interchangeably herein.

[0066] The reconstruction path (shown by the dashed connecting line) can be used to ensure that the encoder 400 and (hereinafter referred to as Figure 5 The decoder 500 described above uses the same reference frame to decode the compressed bit stream 420. The reconstruction path performs the same Figure 5 Functions similar to those performed during the decoding process described above include dequantizing the quantized transform coefficients at the dequantization stage 410 and inverse transforming the dequantized transform coefficients at the inverse transform stage 412 to produce a derivative residual block (also referred to as derivative residual).

[0067] At the reconstruction stage 414, the prediction block predicted at the intra / inter prediction stage 402 may be added to the derivative residual to create a reconstructed block. The loop filtering stage 416 may apply an in-loop filter or other filter to the reconstructed block to reduce distortion such as blocking artifacts. Examples of filters that may be applied at the loop filtering stage 416 include, but are not limited to: a deblocking filter as in AVC, HEVC, VP9, ​​and AV1; a constrained directional enhancement filter as in AV1; and a loop recovery filter as in AV1.

[0068] Other variations of encoder 400 may be used to encode compressed bitstream 420. In some embodiments, a non-transform based encoder may directly quantize the residual signal for certain blocks or frames without transform stage 404. In some embodiments, the encoder may combine quantization stage 406 and dequantization stage 410 into a common stage.

[0069] Figure 5 2 is a block diagram of an example of a decoder 500. The decoder 500 may be implemented in the receiving station 106, for example, by providing a computer software program stored in the memory 204. The computer software program may include machine instructions that, when executed by a processor such as the processor 202, cause the receiving station 106 to receive the received signal in the form of a decoder 500. Figure 5 The decoder 500 may also be implemented in hardware included in the sending station 102 or the receiving station 106, for example.

[0070] Similar to the reconstruction path of the encoder 400 discussed above, the decoder 500, in one example, includes the following stages that perform various functions to produce an output video stream 516 from the compressed bitstream 420: an entropy decoding stage 502, a dequantization stage 504, an inverse transform stage 506, an intra / inter prediction stage 508, a reconstruction stage 510, a loop filtering stage 512, and a post filter stage 514. Other structural variations of the decoder 500 may be used to decode the compressed bitstream 420.

[0071] When the compressed bitstream 420 is presented for decoding, the data elements within the compressed bitstream 420 may be decoded by the entropy decoding stage 502 to produce a set of quantized transform coefficients. The dequantization stage 504 dequantizes the quantized transform coefficients (e.g., by multiplying the quantized transform coefficients by a quantizer value), and the inverse transform stage 506 inverse transforms the dequantized transform coefficients to produce a derivative residual, which may be the same as the derivative residual created by the inverse transform stage 412 in the encoder 400. Using the header information decoded from the compressed bitstream 420, the decoder 500 may use the intra / inter prediction stage 508 to create the same prediction block as was created in the encoder 400 (e.g., at the intra / inter prediction stage 402).

[0072] At reconstruction stage 510, the prediction block may be added to the derivative residual to create a reconstructed block. A loop filtering stage 512 may be applied to the reconstructed block to reduce blocking artifacts. Examples of filters that may be applied at loop filtering stage 512 include, but are not limited to: a deblocking filter as in AVC, HEVC, VP9, ​​and AV1; a constrained directional enhancement filter as in AV1; and a loop restoration filter as in AV1.

[0073] Other filtering may be applied to the reconstructed blocks. In this example, a post-filter stage 514 is applied to the reconstructed blocks to reduce block distortion, and the result is output as an output video stream 516. The output video stream 516 may also be referred to as a decoded video stream, and these terms will be used interchangeably herein.

[0074] Other variations of decoder 500 may be used to decode compressed bitstream 420. In some implementations, decoder 500 may produce output video stream 516 without post-filter stage 514.

[0075] Figure 6 6 is a block diagram of an example of a video stream processing pipeline 600 that uses adaptive filtering to reduce bit rate. The pipeline 600 may include an encoder (e.g., Figure 4 At the beginning of pipeline 600, an input video stream 602 is received. Input video stream 602 may be, for example, Figure 4 The video stream 300 is shown. An input video stream 602 is replicated to obtain a first video stream copy 604 and a second video stream copy 606. Initially, the first video stream copy 604 and the second video stream copy 606 represent identical copies of the input video stream 602.

[0076] In an encoding stage 608 of pipeline 600, first video stream copy 604 is encoded into a reference bitstream 610. Encoding stage 608 may include one or more stages of an encoder (e.g., encoder 400) at which data of first video stream copy 604 is processed. For example, encoding stage 608 may include at least a prediction stage, a transform stage, a quantization stage, and an entropy coding stage (e.g., Figure 4 Intra / inter prediction stage 402, transform stage 404, quantization stage 406 and entropy coding stage 408 are shown).

[0077] Reference bitstream 610 is a compressed bitstream such as Figure 44. The compressed bitstream 420 output by the encoding stage is shown. The first video stream copy 604 is encoded into a reference bitstream 610 using defined transcoding settings, such as a constant quality setting, a constant rate factor setting, or both. The input video stream 602 and the reference bitstream 610 can be compared to determine a distortion value and a bitrate of the reference bitstream 610. The distortion value represents a measure of the amount of distortion introduced into the input video stream by encoding the first copy of the input video stream into the reference bitstream. The bitrate represents a measure of the number of bits used to send the reference bitstream per time unit (e.g., seconds).

[0078] The distortion value of the reference bitstream 610 may be determined using one of a variety of error calculation techniques. For example, the distortion value may be a peak signal-to-noise ratio (PSNR) metric, a mean square error (MSE) metric, a structural similarity index (SSI) metric, or another metric determined using a different error calculation technique. If applicable, conventional methods may be used to determine the PSNR, MSE, SSI, or other distortion metrics disclosed herein.

[0079] At the adaptive filtering stage 612 of the pipeline 600, the second video stream copy 606 is processed. The adaptive filtering stage 612 includes a filter application stage 614, an encoding stage 616, and a bitstream selection stage 618. The filter application stage 614 and the encoding stage 616 include operations repeated for each of a plurality of candidate filters, such as encoding a different copy of the second video stream copy 606 as a candidate bitstream, wherein each candidate bitstream corresponds to and is generated using one of the candidate filters. The bitstream selection stage 618 includes operations performed after the candidate bitstreams are generated, such as selecting one of those candidate bitstreams as an output bitstream 620.

[0080] In the filter application stage 614, the candidate filter is applied to each video frame of the second video stream copy 606. Applying the candidate filter to each video frame of the second video stream copy 606 causes at least some data of the second video stream copy 606 to change. For example, applying the candidate filter may introduce many blurring artifacts within the second video stream copy 606. In another example, applying the candidate filter may introduce a certain amount of noise within the second video stream copy 606. In yet another example, applying the candidate filter may enhance the sharpness of some or all image objects within the second video stream copy 606.

[0081] The candidate filter is a two-dimensional filter of size MxN, where M and N may be the same number or different numbers. Each candidate filter may have the same size. Alternatively, some candidate filters may have sizes that are different from each other. As another alternative, in some cases, each candidate filter may have a different size.

[0082] The candidate filter includes a plurality of coefficients. The specific number of coefficients of the candidate filter depends on the size of the candidate filter. For example, a filter of size 32x32 includes 1,024 coefficients. Each coefficient of the candidate filter has a value ranging from 0 to 1 (including 0 and 1). Therefore, the value of the coefficient of the candidate filter represents a smoothing weight, which is applied to the pixel values ​​within the video frame of the second video stream copy 606.

[0083] Some or all of the candidate filters may be obtained using a nonlinear optimization technique. For example, one or more of a genetic algorithm, a gradient descent algorithm, or another nonlinear optimization technique may be used to obtain some or all of the candidate filters. Alternatively, the candidate filters may be obtained from a list defining candidate filters to be applied to the second video stream copy 606. For example, a lookup table, an object symbol file, or other data source may indicate the candidate filters to be used.

[0084] Applying the candidate filter may result in a degradation in the quality of the second video stream copy 606. Thus, after applying the candidate filter, the resulting filtered second video stream copy has a lower quality than the second video stream copy 606. The degradation in quality caused by applying the filter may be associated with video data that is imperceptible or relatively imperceptible to a human viewer.

[0085] Each candidate filter may correspond to a different type of change made to the data within the second video stream copy 606. In some cases, two or more candidate filters may correspond to the same type of change, but represent different magnitudes or degrees thereof. For example, the first and second filters may both correspond to adding blur artifacts, wherein the first filter adds artifacts of a first blur amount and the second filter adds artifacts of a second blur amount. In another example, the first filter may add a first amount of blur artifacts to each video frame of the second video stream copy 606 and the second filter may add a second amount of blur artifacts to each video frame of the second video stream copy 606.

[0086] In encoding stage 616, after generating a filtered second video stream copy by applying the candidate filter to second video stream copy 606, the filtered second video stream copy is encoded as a candidate bitstream. Encoding stage 616 may include one or more stages of an encoder (e.g., encoder 400) at which data of the filtered second video stream copy is processed. For example, encoding stage 616 may include at least a prediction stage, a transform stage, a quantization stage, and an entropy coding stage (e.g., intra / inter prediction stage 402, transform stage 404, quantization stage 406, and entropy coding stage 408).

[0087] A candidate bitstream is generated for each filtered second video stream copy. The candidate bitstream is a compressed bitstream, such as compressed bitstream 420. The filtered second video stream copy can be encoded as a candidate bitstream using defined transcoding settings, such as a constant quality setting, a constant rate factor setting, or both. Alternatively, the filtered second video stream copy can be encoded as a candidate bitstream using the transcoding settings of the corresponding candidate filter.

[0088] In the bitstream selection stage 618, after the candidate bitstreams are generated, one of those candidate bitstreams is selected as an output bitstream 620, which can be output or stored for further processing (e.g., using a processor such as Figure 5 6. The decoder 500 is shown in FIG. 6. The output bitstream 620 is selected based on the cost value determined for each candidate bitstream.

[0089] To determine the cost value of the candidate bitstream, the candidate bitstream is first compared to the input video stream 602, such as to determine the distortion value and bit rate of the candidate bitstream. The distortion value represents a measure of the amount of distortion introduced to the second video stream copy 606 by applying the corresponding candidate filter and then encoding the filtered second video stream copy as a reference bitstream. The bitrate represents a measure of the number of bits used to send the candidate bitstream per time unit (e.g., seconds). The distortion value of each candidate bitstream can be determined using PSNR, MSE, SSI, or another distortion measurement technique.

[0090] After determining the distortion value and the bit rate of the candidate bitstream, a cost value of the candidate bitstream may be determined based on the difference between the distortion value of the reference bitstream 610 and the distortion value of the candidate bitstream and further based on the difference between the bit rate of the reference bitstream 610 and the bit rate of the candidate bitstream. After determining the cost value for each candidate bitstream, the candidate bitstream having the lowest cost value among the cost values ​​is selected as the output bitstream 620.

[0091] Other variations of 600 may be used to encode the input video stream 602 into an output bitstream 620. As described herein, the candidate bitstreams may be iteratively generated, such as by applying the candidate filters individually to the second video stream copies 606 and then encoding the resulting filtered second video stream copies individually as candidate bitstreams. However, in some embodiments, the candidate bitstreams may be generated after all filtered second video stream copies are generated by the filter application stage 614.

[0092] For example, instead of taking each candidate filter, applying it to the second video stream copy 606, and then encoding the resulting filtered second video stream copy as a candidate bitstream, the filter application stage 614 may produce a filtered second video stream copy for some or all of the candidate filters. After the filter application stage 614 has completed its processing, the encoding stage 616 may receive each filtered second video stream copy and encode each of them as a corresponding candidate bitstream.

[0093] In some embodiments, one or more of the filter application stage 614, the encoding stage 616, or the bitstream selection stage 618 may be external to the adaptive filtering stage 612. For example, the adaptive filtering stage may be limited to applying the candidate filter to a single copy of the second video stream copy 606. In such embodiments, the input to the encoding stage 616 may be the output of the adaptive filtering stage 612.

[0094] In some implementations, the pipeline 600 may include a cost value determination stage separate from the bitstream selection stage 618. For example, the cost value determination stage may perform the operations described above with reference to the bitstream selection stage 618, such as for determining cost values ​​for candidate filters and corresponding candidate bitstreams. The cost value determination stage may be part of the adaptive filtering stage 612, or may occur after the adaptive filtering stage 612 in the pipeline 600.

[0095] In some embodiments, encoding stage 608 and encoding stage 616 may be replaced by a single encoding stage of pipeline 600. For example, the encoding stage may receive as input the first video stream copy 604 and output a reference bitstream 610. Similarly, the encoding stage may receive a filtered second video stream copy produced by filter application stage 614 and output a candidate bitstream, such as for further processing at bitstream selection stage 618.

[0096] Figure 7 is a block diagram of an example of operations performed to select an output bitstream using adaptive filtering. Figure 7 The operations shown and described may be represented in Figure 6 Operations performed at one or more of the filter application stage 614, the encoding stage 616, and / or the bitstream selection stage 618 are shown. Figure 7 The operations shown and described are to encode an input video stream into an output bitstream 700, such as by using a video stream replica 702 representing a replica of the input video stream to generate a candidate bitstream that is selected as the output bitstream 700. For example, the output bitstream 700 and the video stream replica 702 may be, respectively. Figure 6 The output bitstream 620 and the second video stream copy 606 are shown.

[0097] The first portion of the process includes a transform stage 704, a candidate filter application stage 706, and an inverse transform stage 708, which perform operations to produce a filtered video stream copy 710. That is, the first portion of the process is performed to apply the candidate filter to each frame of the video stream copy 702 to produce the filtered video stream copy 710.

[0098] For a given frame of the video stream copy 702, each block of the frame is transformed from the spatial domain to the frequency domain in a transform stage 704. Transforming the given block of the frame may include applying a two-dimensional fast Fourier transform or a similar transform to the pixel values ​​within the block. After the pixel values ​​are transformed to the frequency domain, a candidate filter is applied in a candidate filter application stage 706 to produce filtered frequency domain values. The filtered frequency domain values ​​are then inverse transformed in an inverse transform stage 708 to return those values ​​to the spatial domain. The output of the inverse transform is a filtered video stream copy 710.

[0099] Applying the candidate filter to a given frame of the video stream copy 702 includes dividing the frame into a plurality of overlapping blocks of size PxQ, where P and Q may be the same number or different numbers. The candidate filter is then applied to the overlapping block regions of the frame, such as to denoise those block regions. The overlapping block region overlap may be a half overlap, a quarter overlap, or another amount of overlap. The denoising filter may be sequentially applied to the overlapping block regions of the frame in a raster order or another order.

[0100] For example, for a frame of size 1,024x1,024 and a denoising filter of size 32x32, the filter is first applied to a block region of the frame from pixel locations [0,0] to [312,312]. The next block region of the frame to which the denoising filter is applied is from pixel locations [16,0] to [478,312]. This is repeated until the final application at a block region from pixel locations [992,992] to [10,234,10,234]. Alternatively, the size of the denoising filter and / or the degree of overlap of adjacent block regions can vary with each block region. For example, the degree of overlap can be defined as a specific hysteresis in each independent direction with a parameter value varying between 0 and N-1.

[0101] The second part of the process includes an encoding stage 712 in which the filtered video stream copy 710 is encoded into a candidate bitstream 714. The encoding stage 712 may be, for example, Figure 6 For example, the encoding stage 712 may perform one or more operations for processing the data of the filtered video stream copy 710 to produce a candidate bitstream 714.

[0102] The third portion of the process includes a bitstream comparison stage 716 and a cost value determination stage 718. In the bitstream comparison stage 716, the candidate bitstream 714 is compared to the input video stream to determine a distortion value and a bit rate for the candidate bitstream 714. In the cost value determination stage 718, the distortion value and the bit rate for the candidate bitstream 714 are used to determine a cost value for the candidate bitstream 714 (and therefore for the candidate filter used to generate the candidate bitstream 714).

[0103] For example, the cost value for the candidate bitstream 714 (and therefore for the corresponding candidate filter) may be calculated as:

[0104] E(F)=B(T)-α(D(T′)-D(T))

[0105] where F is the candidate filter, E(F) is the cost value, and T is the reference bitstream (e.g., Figure 6 714 , B(T') is the bit rate of T', D(T') is the distortion value measured for T', D(T) is the distortion value measured for T', and α is the weight applied to the difference between the distortion values ​​D(T') and D(T). The value of α can be defined based on the target bit rate of the video stream copy 702. For example, values ​​of α that result in certain distortion values ​​can be empirically identified.

[0106] In some cases, the difference between the distortion value of the reference bitstream and the distortion value of the candidate bitstream 714 can be nonlinear. That is, whether the difference between those distortion values ​​leads to significant quality or bitrate reduction can depend on the distortion range in which the difference falls. For example, in the case where each of those distortion values ​​is between 30 and 45 decibels, the difference between them is most obvious. A difference of less than 25 decibels can lead to a significant quality loss that has a negative impact on the appearance of the video stream. A difference higher than 50 decibels may result in a quality improvement that is difficult for human viewers to perceive. Therefore, reducing the distortion value while maintaining above 50 decibels can allow for a deeper bitrate reduction. However, since the quality difference is increasingly obvious below 50 decibels, there is less opportunity to reduce the bitrate without affecting the quality of the video stream.

[0107] After determining the cost value for the candidate bitstream 714, determine whether the cost value is the lowest cost value calculated for the candidate bitstream so far. If so, the candidate bitstream 714 is stored as the best candidate bitstream. Otherwise, the candidate bitstream 714 is discarded or otherwise not used further.

[0108] For example, cost values ​​can be iteratively determined for N candidate bitstreams generated using corresponding N candidate filters. After a given cost value is determined for a given candidate bitstream, the cost value can be compared with the stored cost value representing the lowest cost value determined for the N x candidate bitstreams processed so far by the cost value determination stage 718. If the given cost value is lower than the stored cost value, the candidate bitstream is stored to replace the previously stored candidate bitstream. After processing the final (i.e., Nth) candidate bitstream, the stored candidate bitstream is selected as the output bitstream 700.

[0109] Figure 7 Other variations of the process shown in can be used to select the output bitstream 700. In some embodiments, nonlinear optimization techniques can be used to obtain candidate filters. For example, one or more of a genetic algorithm, a gradient descent algorithm, or another nonlinear optimization technique can be used to obtain some or all candidate filters. In some embodiments, optimization operations other than nonlinear processing can be used to obtain candidate filters. For example, a gradient distance function, a Lagrangian function, a linear search function (e.g., using restricted parameters), etc. can be used to obtain candidate filters. For example, using a linear search function, the value of α can be gradually increased from 0 to 1 in increments of 0.1 or other increments. In this way, different candidate bitstreams can be determined for each stepwise increment of α.

[0110]

[0046] Further details of techniques for adaptive filtering of video streams to reduce bit rate are now described. Fig. 8A -B is a flow diagram of an example of a technique 800 for encoding a video stream into an output bitstream using adaptive filtering. Fig. 9 is a flow diagram of an example of a technique 900 for selecting an output bitstream based on cost values ​​determined for candidate filters.

[0111] Technique 800 and / or technique 900 may be implemented, for example, as a software program that may be executed by a computing device such as sending station 102 or receiving station 106. For example, the software program may include machine-readable instructions that may be stored in a memory such as memory 204 or secondary storage 214 and, when executed by a processor such as processor 202, may cause a computing device to perform technique 800 and / or technique 900. Technique 800 and / or technique 900 may be implemented using dedicated hardware or firmware. For example, hardware components are configured to perform technique 800 and / or technique 900. As described above, some computing devices may have multiple memories or processors, and the operations described in technique 800 and / or technique 900 may be distributed using multiple processors, memories, or both.

[0112] For simplicity of explanation, technology 800 and technology 900 are all depicted and described as a series of steps or operations in this article. However, the steps or operations according to the present disclosure can occur in various orders and / or simultaneously. In addition, other steps or operations not presented and described in this article can be used. In addition, all the steps or operations shown may not be needed to implement the technology according to the disclosed subject matter.

[0113] First reference Fig. 8A -B, shows a technique 800 for encoding an input video stream into an output bitstream using adaptive filtering. At 802, a first copy of the input video stream is encoded as a reference bitstream. The first copy of the input video stream is an identical copy of the input video stream. Encoding the first copy of the input video stream into the reference bitstream may include using an encoder (e.g., Figure 4 The encoder 400 shown in FIG. 4 performs prediction, transform, quantization, and entropy coding operations on frames of a first copy of an input video stream.

[0114] At 804, the input video stream and the reference bitstream are compared to determine a distortion value and a bitrate of the reference bitstream. The distortion value represents a measure of the amount of distortion introduced to the input video stream by encoding a first copy of the input video stream into the reference bitstream. The bitrate represents a measure of the number of bits used to send the reference bitstream per unit time (e.g., seconds).

[0115] At 806, the candidate filter is applied to the second copy of the input video stream to generate a filtered second copy of the input video stream. Each filtered second copy of the input video stream corresponds to a single candidate filter used to generate it. Applying the candidate filter to the second copy of the input video stream includes, for each frame of the second copy of the input video stream, first transforming the values ​​within the frame from the spatial domain to the frequency domain. After the transformation, the candidate filter is applied to each block of the frame to generate filtered frequency domain values. The filtered frequency domain values ​​are then inversely transformed from the frequency domain to the spatial domain. The inversely transformed values ​​for each frame represent the values ​​of the filtered second copy of the input video stream. A filtered second copy of the input video stream is generated for some or all of the candidate filters.

[0116] At 808, the filtered second copy of the input video stream is encoded as a candidate bitstream. Encoding the filtered second copy of the input video stream as a candidate bitstream may include using an encoder (e.g., Figure 4 The encoder 400 shown in FIG. 4 is used to perform prediction, transform, quantization, and entropy coding operations on a filtered second copy of an input video stream.

[0117] At 810, a distortion value and a bit rate are determined for each candidate bit stream. The distortion value and the bit rate of the candidate bit stream are determined by comparing the candidate bit stream to the input video stream. The distortion value represents a measure of the amount of distortion introduced to the input video stream by applying the corresponding candidate filter and then encoding the filtered second copy of the input video stream into the candidate bit stream. The bit rate represents a measure of the number of bits used to send the candidate bit stream per time unit (e.g., seconds).

[0118] At 812, a cost value is determined for each candidate bitstream, and therefore for each candidate filter applied to the second copy of the input video stream. The cost value of the candidate bitstream is determined based on the difference between the distortion value of the reference bitstream and the distortion value of the candidate bitstream and further based on the difference between the bitrate of the reference bitstream and the bitrate of the candidate bitstream. In particular, a weighted value is calculated by applying a weight to the difference between the distortion value of the reference bitstream and the distortion value of the candidate bitstream. The cost value of the candidate bitstream can then be determined by subtracting the weighted value from the difference between the bitrate of the reference bitstream and the bitrate of the candidate bitstream.

[0119] At 814, the candidate bitstream having the lowest cost value among the cost values ​​is selected as the output bitstream of the input video stream. Fig. 9 The description includes implementations and examples for selecting an output bitstream by iterating candidate bitstreams.

[0120] At 816, the output bitstream is output or stored. For example, the output bitstream can be sent to a decoder, such as for decoding the output bitstream into an output video stream. In another example, the output bitstream can be stored, such as for later use.

[0121] In some embodiments, some or all of the operations described with respect to technique 800 may be performed for individual candidate filters and then repeated for subsequent candidate filters. For example, operations for applying the candidate filter to a second copy of the input video stream, encoding the filtered second copy of the input video stream as a candidate bitstream, determining a distortion value and a bitrate for the candidate bitstream, and determining a cost value for the candidate bitstream may be performed individually and iteratively for each candidate filter. For example, after the cost value for the Xth candidate bitstream has been determined, the technique may then proceed to apply the X+1th candidate filter to a second copy of the input video stream, and so on.

[0122] Next reference Fig. 9 , a technique 900 for selecting an output bitstream based on a cost value determined for a candidate filter is shown. At 902, a copy of an input video stream is received. For example, the copy of the input video stream may be Figure 7The video stream copy 702 is shown. The copy of the input video stream may represent a copy of the input video stream received at the encoder. Receiving the copy of the input video stream may include, for example, an adaptive filtering stage (e.g., Figure 6 The computational aspect of the encoder, the adaptive filtering stage 612 shown, receives a copy of the input video stream.

[0123] At 904, an Nth filter is applied to each frame of the copy of the input video stream. The Nth filter is one of a number of candidate filters that can be used to filter the copy of the input video stream. Applying the Nth filter to each frame of the copy of the input video stream may include, for each such frame, transforming values ​​within a given frame from a spatial domain to a frequency domain, applying a given candidate filter to each block of the given frame, thereby producing filtered frequency domain values, and then inverse transforming the filtered frequency domain values ​​from the frequency domain to the spatial domain.

[0124] At 906, a cost value is determined for a candidate bitstream produced after applying the Nth filter to the copy of the input video stream. In particular, after applying the Nth filter to the copy of the input video stream, the resulting filtered copy of the input video stream is encoded as a candidate bitstream (e.g., using a method such as Figure 4 The input video stream and the candidate bitstream are then compared to determine a distortion value and a bitrate for the candidate bitstream. A cost value is then determined based on the difference between the distortion value and the bitrate determined for the candidate bitstream and the distortion value and the bitrate determined for the reference bitstream.

[0125] At 908, determine whether the cost value obtained by applying the Nth filter is the minimum cost value determined for the candidate bitstream generated using the N candidate filters. For example, the current minimum cost value obtained by applying the Nth filter to generate the Nth candidate bitstream can be stored in a buffer or other data storage. Determine whether the cost value of the Nth candidate bitstream is the minimum cost value and can include comparing the cost value with the stored cost value. The cost value of the Nth candidate bitstream is the minimum cost value when it is lower than the stored cost value.

[0126] At 910, in response to determining that the cost value of the Nth candidate bitstream is the lowest cost value, the Nth candidate bitstream is stored. Storing the Nth candidate bitstream includes storing the Nth candidate bitstream in a buffer or other data storage to replace the Nth candidate bitstream previously stored therein. For example, the Nth candidate bitstream itself may be stored and / or an index or other indicator of the Nth candidate bitstream may be stored. Alternatively, at 912, in response to determining that the cost value of the Nth candidate bitstream is not the lowest cost value, the Nth candidate bitstream is discarded.

[0127] At 914, it is determined whether the Nth filter is a final candidate filter that is still available for consideration. At 916, in response to determining that the Nth filter is not a final candidate filter that is still available for consideration, the value of N is increased by 1, and the process is repeated. Alternatively, at 918, in response to determining that the Nth filter is a final candidate filter that is still available for consideration, the stored candidate bitstream is selected as the output bitstream. For example, if the Nth candidate bitstream is stored in a buffer or other data storage before determining that the Nth filter is the final remaining filter, the N candidate bitstream is selected as the output bitstream.

[0128] In some embodiments, technique 900 may include comparing one or more aspects of a candidate bitstream to a threshold. For example, the threshold may represent a maximum acceptable quality loss due to applying a corresponding candidate filter and then encoding a filtered copy of the video stream as a candidate bitstream. In another example, the threshold may represent a maximum acceptable latency due to applying a corresponding candidate filter and then encoding a filtered copy of the video stream as a candidate bitstream. In yet another example, the threshold may represent a maximum acceptable computational resource cost due to applying a corresponding candidate filter and then encoding a filtered copy of the video stream as a candidate bitstream.

[0129] In this embodiment, one or more aspects of a candidate bitstream (e.g., quality loss, latency, and / or computational resource cost thereof) are compared to a threshold (e.g., before determining whether the cost value of the candidate bitstream is the lowest cost value, or before storing the candidate bitstream in response to determining that the candidate bitstream has the lowest cost value). In the event that one or more aspects meet the threshold (e.g., by not exceeding a corresponding maximum value), the technique 900 can proceed to continue processing the candidate bitstream. However, in those cases where one or more aspects do not meet the threshold, the candidate bitstream can be discarded or otherwise not processed further.

[0130] The various aspects of encoding and decoding described above illustrate some examples of encoding and decoding techniques.However, it will be understood that encoding and decoding, as those terms are used in the claims, may refer to compression, decompression, transformation or any other processing or change of data.

[0131] The word "example" is used herein to mean used as an example, instance or illustration. Any aspect or design described as an "example" in this article is not necessarily interpreted as being superior to or advantageous over other aspects or designs. On the contrary, the use of the word "example" is intended to present the concept in a specific way. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or the context clearly indicates otherwise, the sentence "X includes A or B" is intended to mean any one of its natural inclusive arrangements. That is, if X includes A; X includes B; or X includes both A and B, then "X includes A or B" is satisfied under any of the above examples. In addition, unless otherwise specified or the context clearly indicates otherwise as being directed to a singular form, the articles "one" and "an" as used in this application and the appended claims should generally be interpreted as meaning "one or more". In addition, the term "embodiment" or the term "one embodiment" used throughout this disclosure should not be intended to mean the same embodiment unless so described.

[0132] The implementation of the sending station 102 and / or the receiving station 106 (and the algorithms, methods, instructions, etc. stored thereon and / or executed thereby, including those executed by the encoder 400 and the decoder 500) can be implemented with hardware, software, or any combination thereof. The hardware may include, for example, a computer, an intellectual property (IP) core, an application specific integrated circuit (ASIC), a programmable logic array, an optical processor, a programmable logic controller, a microcode, a microcontroller, a server, a microprocessor, a digital signal processor, or any other suitable circuit. In the claims, the term "processor" should be understood to cover any one of the above-mentioned hardware individually or in combination. The terms "signal" and "data" can be used interchangeably. In addition, the parts of the sending station 102 and the receiving station 106 do not necessarily have to be implemented in the same manner.

[0133] In addition, in one aspect, for example, a general purpose computer or general purpose processor with a computer program can be used to implement the sending station 102 or the receiving station 106, which, when executed, performs any of the corresponding methods, algorithms and / or instructions described herein. In addition or alternatively, for example, a special purpose computer / processor can be utilized, which can include other hardware for performing any of the methods, algorithms or instructions described herein.

[0134] The sending station 102 and the receiving station 106 can be implemented, for example, on a computer in a video conference system. Alternatively, the sending station 102 can be implemented on a server, and the receiving station 106 can be implemented on a device (such as a handheld communication device) separated from the server. In this example, the sending station 102 can use an encoder 400 to encode the content into a coded video signal and send the coded video signal to the communication device. And then, the communication device can use a decoder 500 to decode the coded video signal. Alternatively, the communication device can decode the content (for example, not sent by the sending station 102) stored locally on the communication device. Other suitable transmission and reception implementation schemes are available. For example, the receiving station 106 can be a usually fixed personal computer rather than a portable communication device, and / or the device including the encoder 400 can also include a decoder 500.

[0135] In addition, all or part of the embodiments of the present disclosure may take the form of a computer program product, for example, accessible from a computer usable medium or a computer readable medium. A computer usable medium or a computer readable medium may be any device that may, for example, tangibly contain, store, transmit, or convey a program for use by any processor or in conjunction with any processor. The medium may be, for example, an electronic, magnetic, optical, electromagnetic, or semiconductor device. Other suitable media are also available.

[0136] The above embodiments and other aspects have been described to facilitate easy understanding of the present disclosure and do not limit the present disclosure. On the contrary, the present disclosure is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which scope should be given the broadest interpretation as permitted under the law so as to include all such modifications and equivalent arrangements.

Claims

1. A method for encoding an input video stream into an output bitstream, the method comprising: encoding a first copy of the input video stream as a reference bitstream; comparing the input video stream and the reference bitstream to determine a first distortion value and a first bit rate of the reference bitstream; For each candidate filter in the plurality of candidate filters: applying the candidate filter to each frame of a second copy of the input video stream to produce a filtered second copy of the input video stream; encoding the filtered second copy of the input video stream as a candidate bitstream; comparing the input video stream and the candidate bitstream to determine a second distortion value and a second bitrate of the candidate bitstream; as well as determining a cost value for the candidate filter based on a difference between the first distortion value and the second distortion value and based on a difference between the first bit rate and the second bit rate; selecting the candidate bitstream corresponding to the candidate filter having the lowest cost value among the cost values ​​as the output bitstream; as well as The output bit stream is output or stored.

2. The method according to claim 1, wherein: The plurality of candidate filters are obtained using at least one of a genetic algorithm, a gradient descent algorithm, and another nonlinear optimization technique.

3. The method according to claim 1, wherein: Applying a given candidate filter to a given frame of a second copy of the input video stream to produce the filtered second copy of the input video stream comprises: transforming the values ​​within the given frame from the spatial domain to the frequency domain; applying the given candidate filter to each block of the given frame after the transform to produce filtered frequency domain values; and The filtered frequency domain values ​​are inversely transformed from the frequency domain to the spatial domain.

4. The method according to claim 3, wherein: Applying the given candidate filter to each block of the given frame to produce filtered frequency domain values ​​comprises: The given candidate filter is used to denoise the overlapping block regions of the given frame.

5. The method according to claim 4, wherein: The overlapping block region is a semi-overlapping block region.

6. The method according to any one of claims 1 to 5, wherein: Determining the cost value for a given candidate filter among the plurality of candidate filters comprises: calculating a weighted value by applying a weight to a difference between the first distortion value and the second distortion value; and The cost value is determined by subtracting the weighted value from a difference between the first bit rate and the second bit rate.

7. The method according to claim 6, wherein: The weights are defined based on a target for the second distortion value.

8. An apparatus for encoding an input video stream into an output bitstream, the apparatus comprising: Memory, and a processor configured to execute instructions stored in the memory for: encoding a first copy of the input video stream as a reference bitstream; comparing the input video stream and the reference bitstream to determine a first distortion value and a first bit rate of the reference bitstream; For each candidate filter in the plurality of candidate filters: applying the candidate filter to each frame of a second copy of the input video stream to produce a filtered second copy of the input video stream; encoding the filtered second copy of the input video stream as a candidate bitstream; comparing the input video stream and the candidate bitstream to determine a second distortion value and a second bitrate of the candidate bitstream; as well as determining a cost value for the candidate filter based on a difference between the first distortion value and the second distortion value and based on a difference between the first bit rate and the second bit rate; selecting the candidate bitstream corresponding to the candidate filter having the lowest cost value among the cost values ​​as the output bitstream; as well as The output bit stream is output or stored.

9. The device according to claim 8, wherein: The plurality of candidate filters are obtained using at least one of a genetic algorithm, a gradient descent algorithm, and another nonlinear optimization technique.

10. The device according to claim 8, wherein: The instructions for applying a given candidate filter to a given frame of a second copy of the input video stream to produce the filtered second copy of the input video stream include instructions for: transforming the values ​​within the given frame from the spatial domain to the frequency domain; applying the given candidate filter to each block of the given frame after the transform to produce filtered frequency domain values; as well as The filtered frequency domain values ​​are inversely transformed from the frequency domain to the spatial domain.

11. The device according to claim 10, wherein: The instructions for applying the given candidate filter to each block of the given frame to produce filtered frequency domain values ​​include instructions for: The given candidate filter is used to denoise the overlapping block regions of the given frame.

12. The device according to claim 11, wherein The overlapping block region is a semi-overlapping block region.

13. The device according to any one of claims 8 to 12, wherein: The instructions for determining the cost value for a given candidate filter in the plurality of candidate filters include instructions for: calculating a weighted value by applying a weight to a difference between the first distortion value and the second distortion value; and The cost value is determined by subtracting the weighted value from a difference between the first bit rate and the second bit rate.

14. The device according to claim 13, wherein: The weights are defined based on a target for the second distortion value.

15. A non-transitory computer-readable storage medium storing instructions, which when executed by a data processing device causes the data processing device to perform operations comprising: encoding a first copy of an input video stream as a reference bitstream; comparing the input video stream and the reference bitstream to determine a first distortion value and a first bit rate of the reference bitstream; For each candidate filter in the plurality of candidate filters: applying the candidate filter to each frame of a second copy of the input video stream to produce a filtered second copy of the input video stream; encoding the filtered second copy of the input video stream as a candidate bitstream; comparing the input video stream and the candidate bitstream to determine a second distortion value and a second bitrate of the candidate bitstream; as well as determining a cost value for the candidate filter based on a difference between the first distortion value and the second distortion value and based on a difference between the first bit rate and the second bit rate; selecting the candidate bitstream corresponding to the candidate filter having the lowest cost value among the cost values ​​as an output bitstream; as well as The output bit stream is output or stored.

16. The non-transitory computer-readable storage medium of claim 15, wherein: The plurality of candidate filters are obtained using at least one of a genetic algorithm, a gradient descent algorithm, and another nonlinear optimization technique.

17. The non-transitory computer-readable storage medium of claim 15, wherein: Applying a given candidate filter to a given frame of the second copy of the input video stream to produce the filtered second copy of the input video stream comprises: transforming the values ​​within the given frame from the spatial domain to the frequency domain; applying the given candidate filter to each block of the given frame after the transform to produce filtered frequency domain values; and The filtered frequency domain values ​​are inversely transformed from the frequency domain to the spatial domain.

18. The non-transitory computer-readable storage medium of claim 17, wherein: Applying the given candidate filter to each block of the given frame to produce the filtered frequency domain values ​​comprises: The given candidate filter is used to denoise the overlapping block regions of the given frame.

19. The non-transitory computer-readable storage medium of claim 18, wherein: The overlapping block region is a semi-overlapping block region.

20. The non-transitory computer-readable storage medium according to any one of claims 15 to 19, wherein: Determining the cost value for a given candidate filter among the plurality of candidate filters comprises: calculating a weighted value by applying a weight to a difference between the first distortion value and the second distortion value; and determining the cost value by subtracting the weighted value from a difference between the first bit rate and the second bit rate, Wherein, the weight is defined based on a target of the second distortion value.

Images