System and method for smoothing mode prediction

Abstract

Various implementations described herein include methods and systems for video encoding and decoding. In one aspect, a method of video decoding includes receiving video data including a first block from a video bitstream, where the first block is encoded in a smooth mode. The method further includes identifying a set of reference samples for the first block and deriving a first prediction value for the first block. The method further includes deriving a refined first prediction value for the first block using a weighted sum of a first reference sample of the set of reference samples and the first prediction value, and decoding the first block based on the refined first prediction value.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 464,155, filed on May 4, 2023, entitled “Smooth Pattern with Separable Boundary Filtering,” and this application is a continuation of and claims priority to U.S. Patent Application No. 18 / 474,517, filed on September 26, 2023, entitled “Systems and Methods for Smooth Pattern Prediction.” Technical Field

[0003] The disclosed embodiments relate generally to video coding and decoding, including but not limited to systems and methods for smooth mode prediction for video coding / decoding. Background Art

[0004] Digital video is supported by a variety of electronic devices, such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smart phones, video teleconferencing devices, video streaming devices, etc. Electronic devices send and receive or otherwise transmit digital video data across communication networks and / or store digital video data on storage devices. Due to the limited bandwidth capacity of communication networks and the limited storage resources of storage devices, video codecs may be used to compress video data according to one or more video codec standards before transmitting or storing the video data.

[0005] A variety of video codec standards have been developed. For example, video codec standards include Alliance for Open Media Video 1 (AOMedia Video 1 (AV1)), Versatile Video Codec (VVC), Joint Exploration Test Model (JEM), High Efficiency Video Coding (HEVC / H.265), Advanced Video Codec (AVC / H.264), and Moving Picture Experts Group (MPEG) coding. Video codecs typically use prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that exploit inherent redundancy in video data. Video codecs are designed to compress video data into a form that uses a lower bit rate while avoiding or reducing degradation in video quality.

[0006] HEVC, also known as H.265, is a video compression standard designed as part of the MPEG-H project. The HEVC / H.265 standard was published by ITU-T in 2013 (version 1), ISO / IEC in 2014 (version 2), 2015 (version 3), and 2016 (version 4). Versatile Video Codec (VVC), also known as H.266, is a video compression standard designed to succeed HEVC. The VVC / H.266 standard was published by ITU-T in 2020 (version 1) and ISO / IEC in 2022 (version 2). AV1 is an open video codec format designed to replace HEVC. On January 8, 2019, the validated version 1.0.0 was released, which contains errata 1 of the specification. Summary of the invention

[0007] As described in more detail below, smooth prediction modes in current systems (e.g., SMOOTH, SMOOTH-H, and SMOOTH-V in AV1) do not take into account the uneven distribution of available reference samples (e.g., only upper and left reference samples are available). The systems and methods described herein improve prediction accuracy by biasing (weighting) prediction using upper and left reference samples.

[0008] According to some embodiments, a video decoding method is provided. The method includes: (i) receiving video data including a plurality of blocks from a video bitstream, the plurality of blocks including a first block, wherein the first block is encoded in one of a plurality of smoothing modes (e.g., SMOOTH-B, SMOOTH-H, or SMOOTH-V mode); (ii) identifying a reference sample set for the first block; (iii) deriving a first prediction value for the first block; (iv) deriving a refined first prediction value for the first block using a weighted sum of a first reference sample of the reference sample set and the first prediction value; and (v) decoding the first block based on the refined first prediction value.

[0009] According to some embodiments, a video encoding method is provided. The method includes: (i) receiving video data including a plurality of blocks, the plurality of blocks including a first block, wherein the first block is to be encoded in one of a plurality of smoothing modes; (ii) identifying a reference sample set for the first block; (iii) deriving a first prediction value for the first block; (iv) deriving a refined first prediction value for the first block using a weighted sum of a first reference sample of the reference sample set and the first prediction value; and (v) encoding the first block based on the refined first prediction value.

[0010] According to some embodiments, a computing system is provided, such as a streaming system, a server system, a personal computing system, or other electronic device. The computing system includes a control circuit and a memory storing one or more groups of instructions. One or more groups of instructions include instructions for performing any method described herein. In some embodiments, the computing system includes an encoder component and a decoder component (e.g., a transcoder component).

[0011] According to some embodiments, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores one or more sets of instructions executed by a computing system. The one or more sets of instructions include instructions for performing any of the methods described herein.

[0012] Therefore, methods, devices and systems for encoding and decoding video are disclosed. Such methods, devices and systems can supplement or replace traditional methods, devices and systems for video encoding / decoding.

[0013] The features and advantages described in the specification are not necessarily all inclusive, and in particular, some additional features and advantages will be apparent to those of ordinary skill in the art in view of the drawings, descriptions, and claims provided in this disclosure. In addition, it should be noted that the language used in the specification is primarily selected for readability and instructional purposes, and is not necessarily intended to describe or limit the subject matter described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to understand the present disclosure in more detail, a more specific description may be made by reference to the features of various embodiments, some of which are shown in the accompanying drawings. However, the attached drawings only illustrate the relevant features of the present disclosure and are therefore not necessarily considered to be limiting, as those skilled in the art will understand after reading the present disclosure that the specification may admit other effective features.

[0015] Figure 1 is a block diagram illustrating an exemplary communication system in accordance with some embodiments.

[0016] Figure 2A is a block diagram illustrating exemplary elements of an encoder assembly according to some embodiments.

[0017] Figure 2B is a block diagram illustrating exemplary elements of a decoder component according to some embodiments.

[0018] Figure 3 is a block diagram illustrating an exemplary server system in accordance with some embodiments.

[0019] Figures 4A to 4D An exemplary codec tree structure according to some embodiments is shown.

[0020] Figure 5A Examples of directional intra prediction mode angles are shown in accordance with some embodiments.

[0021] Figures 5B to 5C Exemplary reference samples of a current block according to some embodiments are shown.

[0022] Figure 5D Exemplary weightings for smoothing modes according to some embodiments are shown.

[0023] Figure 5E Exemplary reference samples of a current block according to some embodiments are shown.

[0024] Fig. 6A is a flow chart illustrating an exemplary method of video encoding according to some embodiments.

[0025] Figure 6B is a flow chart illustrating an exemplary method of video decoding according to some embodiments.

[0026] As is common practice, the various features shown in the drawings are not necessarily drawn to scale, and like reference numerals may be used throughout the specification and drawings to denote like features. DETAILED DESCRIPTION

[0027] Among other things, the present disclosure describes a new smooth prediction mode that can improve prediction accuracy (compared to previous smooth modes) by using upper and / or left reference samples to bias prediction. For example, to predict a sample in the current block using spatially neighboring reference samples, a horizontal prediction value P may be derived. H , then use the left reference sample and P H The weighted sum of the refined horizontal prediction value P H '. As another example, the vertical prediction value P is obtained V , then use the reference sample and P V The weighted sum of the obtained fine-scale horizontal prediction value P V '. In this way, the encoding and decoding accuracy can be improved (e.g., reducing artifacts in the decoded video) compared to other smooth prediction modes.

[0028] Exemplary systems and devices are described below.

[0029] Figure 11 is a block diagram illustrating an exemplary communication system 100 according to some embodiments. The communication system 100 includes a source device 102 and a plurality of electronic devices 120 (e.g., electronic device 120-1 to electronic device 120-m) that are communicatively coupled to each other via one or more networks. In some embodiments, the communication system 100 is a streaming media system, such as for video-enabled applications, such as video conferencing applications, digital television applications, and media storage and / or distribution applications.

[0030] Source device 102 includes a video source 104 (e.g., a camera component or a media storage device) and an encoder component 106. In some embodiments, video source 104 is a digital camera (e.g., configured to create an uncompressed video sample stream). Encoder component 106 generates one or more coded video bitstreams based on the video stream. The video stream from video source 104 can be high data volume compared to the coded video bitstream 108 generated by encoder component 106. Because the coded video bitstream 108 has a lower data volume (less data) than the video stream from the video source, the coded video bitstream 108 requires less transmission bandwidth and less storage space than the video stream from video source 104. In some embodiments, source device 102 does not include encoder component 106 (e.g., configured to send uncompressed video data to network 110).

[0031] One or more networks 110 represent any number of networks that transmit information between source device 102, server system 112, and / or electronic device 120, including, for example, wired and / or wireless communication networks. One or more networks 110 can exchange data in circuit switching and / or packet switching channels. Representative networks include telecommunication networks, local area networks, wide area networks, and / or the Internet.

[0032] One or more networks 110 include a server system 112 (e.g., a distributed / cloud computer system). In some embodiments, the server system 112 is (or includes) a streaming media server (e.g., configured to store and / or distribute video content, such as an encoded video stream from a source device 102). The server system 112 includes a codec component 114 (e.g., configured to encode and / or decode video data). In some embodiments, the codec component 114 includes an encoder component and / or a decoder component. In various embodiments, the encoder component 114 is instantiated as hardware, software, or a combination thereof. In some embodiments, the codec component 114 is configured to decode the encoded video bitstream 108 and re-encode the video data using different encoding standards and / or methods to generate encoded video data 116. In some embodiments, the server system 112 is configured to generate multiple video formats and / or encodings based on the encoded video bitstream 108.

[0033] In some embodiments, the server system 112 functions as a media aware network element (MANE). For example, the server system 112 may be configured to prune the encoded video bitstream 108 so that a potentially different bitstream is dedicated to one or more electronic devices 120. In some embodiments, a different MANE than the server system 112 is provided.

[0034] The electronic device 120-1 includes a decoder component 122 and a display 124. In some embodiments, the decoder component 122 is configured to decode the encoded video data 116 to generate an output video stream that can be presented on a display or other type of presentation device. In some embodiments, one or more electronic devices 120 do not include a display component (e.g., communicatively coupled to an external display device and / or include a media storage device). In some embodiments, the electronic device 120 is a streaming media client. In some embodiments, the electronic device 120 is configured to access the server system 112 to obtain the encoded video data 116.

[0035] The source device and / or multiple electronic devices 120 are sometimes referred to as “end devices” or “user devices.” In some embodiments, the source device 102 and / or one or more electronic devices 120 are examples of server systems, personal computers, portable devices (e.g., smartphones, tablets, or laptops), wearable devices, video conferencing devices, and / or other types of electronic devices.

[0036] In an example operation of the communication system 100, the source device 102 sends a coded video bitstream 108 to a server system 112. For example, the source device 102 can encode and decode a picture stream captured by the source device. The server system 112 receives the coded video bitstream 108 and can decode and / or encode the coded video bitstream 108 using a codec component 114. For example, the server system 112 can encode the video data to be more suitable for network transmission and / or storage. The server system 112 can send the coded video data 116 (e.g., one or more coded video bitstreams) to one or more electronic devices 120. Each electronic device 120 can decode the coded video data 116 to recover and optionally display the video picture.

[0037] Figure 2A1 is a block diagram illustrating exemplary elements of an encoder component 106 according to some embodiments. The encoder component 106 receives a source video sequence from a video source 104. In some embodiments, the encoder component includes a receiver (e.g., a transceiver) component configured to receive the source video sequence. In some embodiments, the encoder component 106 receives the video sequence from a remote video source (e.g., a video source that is a component of a device different from the encoder component 106). The video source 104 can provide the source video sequence in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8 bits, 10 bits, or 12 bits), any color space (e.g., television broadcast (Broadcasting Service television) BT.601 color space (Y CrCB) or red, green, and blue (RGB)) and any suitable sampling structure (e.g., Y CrCb4:2:0 or Y CrCb4:4:4). In some embodiments, the video source 104 is a storage device that stores pre-captured or prepared videos. In some embodiments, the video source 104 is a camera that captures local image information as a video sequence. Video data may be provided as a plurality of individual pictures that convey motion when viewed sequentially. The pictures themselves may be organized as a spatial array of pixels, where each pixel may include one or more samples depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be readily understood by one of ordinary skill in the art. The following description focuses on samples.

[0038] The encoder component 106 is configured to encode and / or compress the pictures of the source video sequence into an encoded video sequence 216 in real time or under other time constraints required by the application. Implementing the appropriate encoding speed is a function of the controller 204. In some embodiments, the controller 204 controls other functional units as described below and is functionally coupled to the other functional units. The parameters set by the controller 204 may include rate control related parameters (e.g., picture skipping, quantizer, lambda value, etc. for rate distortion optimization techniques), picture size, picture group (GOP) layout, maximum motion vector search range, etc. Other functions of the controller 204 can be easily identified by those of ordinary skill in the art as they may belong to the encoder component 106 optimized for a particular system design.

[0039] In some embodiments, the encoder component 106 is configured to operate in an encoding loop. In a simplified example, the encoding loop includes a source encoder 202 (e.g., for creating symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder 210. The decoder 210 reconstructs the symbols to create sample data in a manner similar to that of the (remote) decoder (when the compression between the symbols and the encoded video bitstream is lossless). The reconstructed sample stream (sample data) is input into a reference image memory 208. Since the decoding of the symbol stream results in a bit-accurate result that is independent of the decoder location (local or remote), the content in the reference image memory 208 is also bit-accurate between the local encoder and the remote encoder. In this way, the prediction section of the encoder will understand that the reference picture samples have the same sample values ​​as interpreted by the decoder when using the prediction during decoding. The principles of reference image synchronization (and the drift resulting therefrom if the synchronization cannot be maintained, such as due to channel errors) are known to those of ordinary skill in the art.

[0040] The operation of the decoder 210 may be the same as that of the remote decoder, for example, the decoder component 122, which will be described below in conjunction with Figure 2B However, simply refer to Figure 2B , since symbols are available and the encoding / decoding of symbols of the encoded video sequence by the entropy encoder 214 and the parser 254 can be lossless, the entropy decoding portion of the decoder component 122, including the buffer memory 252 and the parser 254, may not be fully implemented in the local decoder 210.

[0041] Except for parsing / entropy decoding, the decoder techniques described herein can exist in a corresponding encoder in substantially the same functional form. Therefore, the disclosed subject matter focuses on the decoder operation. The description of the encoder techniques can be abbreviated because the encoder techniques may be opposite to the decoder techniques. Only in certain areas is a more detailed description required and is provided below.

[0042] As part of its operation, source encoder 202 may perform motion compensated predictive coding, which predictively encodes an input frame with reference to one or more previously encoded frames in a video sequence (designated as reference frames). In this manner, encoding engine 212 encodes the difference between a pixel block of an input frame and a pixel block of a reference frame, which may be selected as a predictive reference for the input frame. Controller 204 may manage the encoding operations of source encoder 202, including, for example, the setting of parameters and subgroup parameters for encoding video data.

[0043] The decoder 210 decodes the encoded video data of the frame that can be designated as the reference frame based on the symbols created by the source encoder 202. The operation of the encoding engine 212 can advantageously be a lossy process. When the encoded and decoded video data is in the video decoder ( Figure 2A The decoder 210 replicates the decoding process that may be performed on the reference frame by the remote video decoder and may cause the reconstructed reference frame to be stored in the reference image memory 208. In this way, the encoder component 106 locally stores a copy of the reconstructed reference frame having common content as the reconstructed reference frame to be obtained by the remote video decoder (without transmission errors).

[0044] The predictor 206 may perform a prediction search for the encoding engine 212. That is, for a new frame to be encoded, the predictor 206 may search the reference image memory 208 for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may be used as appropriate prediction references for the new picture. The predictor 206 may operate on a sample block-by-pixel block basis to find an appropriate prediction reference. In some cases, as determined by the search results obtained by the predictor 206, the input picture may have a prediction reference extracted from a plurality of reference pictures stored in the reference image memory 208.

[0045] The outputs of all of the above functional units may be entropy encoded in entropy encoder 214. Entropy encoder 214 translates the symbols generated by the various functional units into an encoded video sequence by losslessly compressing the symbols according to techniques known to those of ordinary skill in the art (e.g., Huffman coding, variable length coding, and / or arithmetic coding).

[0046] In some embodiments, the output of the entropy encoder 214 is coupled to a transmitter. The transmitter may be configured to buffer the coded video sequences created by the entropy encoder 214 to prepare them for transmission via a communication channel 218, which may be a hardware / software link to a storage device storing the coded video data. The transmitter may be configured to merge the coded video data from the source encoder 202 with other data to be sent, such as coded audio data and / or an auxiliary data stream (source not shown). In some embodiments, the transmitter may send additional data with the coded video. The source encoder 202 may include such data as part of the coded video sequence. The additional data may include time / space / SNR enhancement layers, other forms of redundant data (e.g., redundant pictures and slices), supplemental enhancement information (SEI) messages, visual usability information (VUI) parameter set fragments, and the like.

[0047] The controller 204 can manage the operation of the encoder component 106. During encoding, the controller 204 can assign a specific coded picture type to each coded picture, which can affect the coding technology applied to each picture. For example, a picture can be designated as an intra picture (I picture), a predicted picture (P picture), or a bidirectional predicted picture (B picture). Intra-frame pictures can be encoded and decoded without using any other frame in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, independent decoder refresh (IDR) pictures. Those of ordinary skill in the art know those variants of I pictures and their respective applications and features, so they are not repeated here. Predicted pictures can be encoded and decoded using intra-frame prediction or inter-frame prediction, using at most one motion vector and reference index to predict the sample values ​​of each block. Bidirectional predicted pictures can be encoded and decoded using intra-frame prediction or inter-frame prediction, using at most two motion vectors and reference indexes to predict the sample values ​​of each block. Similarly, multi-predicted pictures can use more than two reference pictures and associated metadata to reconstruct a single block.

[0048] The source picture may typically be spatially subdivided into a number of blocks of samples (e.g. blocks of 4x4, 8x8, 4x8 or 16x16 samples each) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding allocation applied to the block's corresponding picture. For example, blocks of an I picture may be non-predictively coded, or they may be predictively coded (spatial prediction or intra prediction) with reference to already coded blocks of the same picture. Pixel blocks of P pictures may be non-predictively coded by spatial prediction or by temporal prediction with reference to one previously coded reference picture. B image blocks may be non-predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.

[0049] Video can be captured as multiple source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often abbreviated as intra-frame prediction) exploits spatial correlations in a given image, and inter-picture prediction exploits (temporal or other) correlations between pictures. In one example, a particular picture under encoding / decoding, referred to as the current picture, is divided into blocks. When a block in the current picture is similar to a reference block in a reference picture that has been previously encoded and still cached in the video, the block in the current picture can be encoded by a vector called a motion vector. In the case of using multiple reference pictures, the motion vector points to a reference block in a reference picture and can have a third dimension that identifies the reference picture.

[0050] The encoder component 106 can perform encoding operations according to a predetermined video encoding technique or standard, such as any of the techniques or standards described herein. In its operation, the encoder component 106 can perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video encoding technique or standard being used.

[0051] Figure 2B is a block diagram illustrating exemplary elements of decoder component 122 according to some embodiments. Figure 2B The decoder component 122 in is coupled to the channel 218 and the display 124. In some embodiments, the decoder component 122 includes a transmitter coupled to the loop filter 256 and configured to send data to the display 124 (eg, via a wired or wireless connection).

[0052] In some embodiments, the decoder component 122 includes a receiver coupled to the channel 218 and configured to receive data from the channel 218 (e.g., via a wired or wireless connection). The receiver can be configured to receive one or more coded video sequences to be decoded by the decoder component 122. In some embodiments, the decoding of each coded video sequence is independent of other coded video sequences. Each coded video sequence can be received from the channel 218, which can be a hardware / software link to a storage device storing coded video data. The receiver can receive coded video data and other data, such as coded audio data and / or auxiliary data streams, which can be forwarded to their respective entities (not shown). The receiver can separate the coded video sequence from the other data. In some embodiments, the receiver receives coded video and additional (redundant) data. The additional data can be included as part of the coded video sequence. The decoder component 122 can use the additional data to decode the data and / or more accurately reconstruct the original video data. The additional data can be in the form of, for example, time, space or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0053] According to some embodiments, the decoder component 122 includes a buffer memory 252, a parser 254 (sometimes also referred to as an entropy decoder), a scaler / inverse transform unit 258, an intra prediction unit 262, a motion compensated prediction unit 260, an aggregator 268, a loop filter unit 256, a reference image memory 266, and a current image memory 264. In some embodiments, the decoder component 122 is implemented as an integrated circuit, a series of integrated circuits, and / or other electronic circuits. In some embodiments, the decoder component 122 is at least partially implemented in software.

[0054] Buffer memory 252 is coupled between channel 218 and parser 254 (e.g., to combat network jitter). In some embodiments, buffer memory 252 is separate from decoder component 122. In some embodiments, a separate buffer memory is provided between the output of channel 218 and decoder component 122. In some embodiments, in addition to buffer memory 252 internal to decoder component 122 (e.g., which is configured to handle playout timing), a separate buffer memory is provided external to decoder component 122 (e.g., to combat network jitter). Buffer memory 252 may not be required, or may be small, when receiving data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network. For use over a best-effort packet network such as the Internet, buffer memory 252 may be necessary, may be relatively large and may advantageously have an adaptive size, and may be implemented at least in part in an operating system or similar element (not depicted) external to decoder component 122.

[0055] The parser 254 is configured to reconstruct the symbols 270 from the coded video sequence. The symbols may include, for example, information for managing the operation of the decoder component 122, and / or information for controlling a rendering device such as a display 124. The control information for the rendering device may be in the form of, for example, a supplemental enhancement information (SEI) message or a video usability information (VUI) parameter set fragment (not shown). The parser 254 parses (entropy decodes) the coded video sequence. The encoding of the coded video sequence may be based on a video coding technique or standard, and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser 254 may extract a set of subgroup parameters of at least one subset of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the grouping. The subgroup may include a group of pictures (GOP), a picture, a slice, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), and the like. The parser 254 may also extract information such as transform coefficients, quantizer parameter values, motion vectors, and the like from the coded video sequence.

[0056] Depending on the type of the coded video picture or its portion (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors, the reconstruction of the symbol 270 may involve multiple different units. Which units are involved and how they are involved may be controlled by sub-group control information parsed from the coded video sequence by the parser 254. For clarity, such sub-group control information flow between the parser 254 and the multiple units is not described below.

[0057] The decoder component 122 may be conceptually subdivided into a plurality of functional units, and in some embodiments, these units closely interact with each other and may be at least partially integrated with each other. However, for the sake of clarity, the conceptual subdivision of functional units is maintained here.

[0058] The sealer / inverse transform unit 258 receives the quantized transform coefficients and control information (e.g., which transform to use, block size, quantization factor, and / or quantization scaling matrix) from the parser 254 as symbols 270. The sealer / inverse transform unit 258 may output a block including sample values ​​that may be input into the aggregator 268.

[0059] In some cases, the output samples of the scaler / inverse transform unit 258 belong to intra-coded blocks, i.e., blocks that do not use prediction information from previously reconstructed pictures, but can use prediction information from previously reconstructed portions of the current picture. Such prediction information may be provided by the intra-prediction unit 262. The intra-prediction unit 262 may use surrounding reconstructed information obtained from the current (partially reconstructed) picture from the current picture memory 264 to generate a block of the same size and shape as the block being reconstructed. The aggregator 268 may add the prediction information already generated by the intra-prediction unit 262 to the output sample information provided by the scaler / inverse transform unit 258 on a per-sample basis.

[0060] In other cases, the output samples of the scaler / inverse transform unit 258 belong to blocks that are mutually encoded and possibly motion-compensated. In this case, the motion compensated prediction unit 260 may access the reference picture memory 266 to obtain samples for prediction. After the extracted samples are motion compensated according to the symbol 270 belonging to the block, these samples may be added to the output of the scaler / inverse transform unit 258 (in this case referred to as residual samples or residual signals) by the aggregator 268 to generate output sample information. The address within the reference picture memory 266 from which the motion compensated prediction unit 260 extracts the predicted samples may be controlled by a motion vector. The motion vector is available to the motion compensated prediction unit 260 in the form of a symbol 270, which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​obtained from the reference picture memory 266, motion vector prediction mechanisms, etc., when sub-sample accurate motion vectors are used.

[0061] The output samples of aggregator 268 may be subjected to various loop filtering techniques in loop filter unit 256. The video compression techniques may include loop filter techniques that are controlled by parameters included in the encoded video bitstream, which are available to loop filter unit 256 as symbols 270 in parser 254, but may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of a coded picture or coded video sequence, and to previously reconstructed and loop filtered sample values.

[0062] The output of the loop filter unit 256 may be a sample stream that may be output to a rendering device such as the display 124 and stored in the reference image memory 266 for use in future inter-frame prediction.

[0063] Certain coded pictures, once fully reconstructed, may be used as reference images for future predictions. Once a coded picture is fully reconstructed and the coded picture has been identified (e.g., by parser 254) as a reference picture, the current reference picture may become part of reference picture memory 266, and a new current picture memory may be reallocated before starting reconstruction of the next coded picture.

[0064] The decoder component 122 can perform decoding operations according to a predetermined video compression technology, which can be recorded in a standard, such as any standard described herein. The encoded video sequence can conform to the syntax specified by the video compression technology or standard being used, in the sense that the encoded video sequence conforms to the syntax of the video compression technology or standard, as specified in the video compression technology document or standard, particularly in the profile document therein. In addition, in order to comply with some video compression technologies or standards, the complexity of the encoded video sequence can be within the range defined by the level of the video compression technology or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (e.g., measured in megasamples per second), maximum reference image size, etc. In some cases, the limits set by the level can be further limited by the assumed reference decoder (HRD) specification and the metadata of the HRD buffer management signaled in the encoded video sequence.

[0065] Figure 3306, and a plurality of communication buses 312 for interconnecting these components. In some embodiments, the control circuit 302 includes one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a data processing unit (DPU)). In some embodiments, the control circuit includes one or more field programmable gate arrays (FPGAs), hardware accelerators, and / or one or more integrated circuits (e.g., application specific integrated circuits).

[0066] The network interface 304 can be configured to connect to one or more communication networks (e.g., wireless, wired and / or optical networks). The communication network can be a local area network, a wide area network, a metropolitan area network, an in-vehicle and industrial network, a real-time network, a delay-tolerant network, etc. Examples of communication networks include local area networks, such as local area networks, wireless local area networks, cellular networks including global mobile communications (GSM), third generation (3G), fourth generation (4G), fifth generation (5G), long-term evolution (LTE), etc., television wired or wireless wide area digital networks including cable TV, satellite TV and terrestrial broadcast television, in-vehicle and industrial networks including control area network (CAN) buses, etc. Such communication can be one-way reception only (e.g., broadcast television), one-way transmission only (e.g., a CAN bus to certain CAN bus devices) or two-way (e.g., using a local or wide area digital network to other computing systems). Such communication can include communication to one or more cloud computing networks.

[0067] The user interface 306 includes one or more output devices 308 and / or one or more input devices 310. The input devices 310 may include one or more of a keyboard, a mouse, a trackpad, a touch screen, a data glove, a joystick, a microphone, a scanner, a camera, etc. The output devices 308 may include one or more of an audio output device (e.g., a speaker), a visual output device (e.g., a display or a display), etc.

[0068] The memory 314 may include high-speed random access memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM), double rate random access memory (DDR RAM), and / or other random access solid-state memory devices) and / or non-volatile memory (e.g., one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, and / or other non-volatile solid-state storage devices). The memory 314 may optionally include one or more storage devices remote from the control circuit 302. The memory 314, or alternatively, the non-volatile solid-state memory device within the memory 314, includes a non-transitory computer-readable storage medium. In some embodiments, the memory 314 or the non-transitory computer-readable storage medium of the memory 314 stores the following programs, modules, instructions, and data structures, or a subset or superset thereof: operating system 316, network communication module 318, codec module 320, picture memory 352.

[0069] The operating system 316 includes programs for handling various basic system services and for performing hardware-dependent tasks.

[0070] The network communication module 318 is used to connect the server system 112 to other computing devices through the one or more network interfaces 304 (eg, via wired and / or wireless connections).

[0071] The codec module 320 is used to perform various functions with respect to encoding and / or decoding data (e.g., video data). In some embodiments, the codec module 320 is an instance of the codec component 114. The codec module 320 includes, but is not limited to, one or more of the following: a decoding module 322 and an encoding module 340.

[0072] The decoding module 322 is used to perform various functions with respect to decoding encoded data, such as the functions previously described with respect to the decoder component 122 .

[0073] The encoding module 340 is used to perform various functions with respect to encoded data, such as those previously described with respect to the encoder component 106 .

[0074] Picture memory 352 is used to store pictures and picture data, such as for use with codec module 320. In some embodiments, picture memory 352 includes one or more of reference picture memory 208, buffer memory 252, current picture memory 264, and reference picture memory 266.

[0075] In some embodiments, the decoding module 322 includes a parsing module 324 (e.g., configured to perform the various functions previously described with respect to the parser 254), a transform module 326 (e.g., configured to perform the various functions previously described with respect to the scaler / inverse transform unit 258), a prediction module 328 (e.g., configured to perform the various functions previously described with respect to the motion compensated prediction unit 260 and / or the intra-frame prediction unit 262), and a filter module 330 (e.g., configured to perform the various functions previously described with respect to the loop filter 256).

[0076] In some embodiments, the encoding module 340 includes a code module 342 (e.g., configured to perform various functions previously described with respect to the source encoder 202 and / or the encoding engine 212) and a prediction module 344 (e.g., configured to perform various functions previously described with respect to the predictor 206). In some embodiments, the decoding module 322 and / or the encoding module 340 includes Figure 3 A subset of the modules shown. For example, both the decoding module 322 and the encoding module 340 use a shared prediction module.

[0077] Each of the above modules stored in the memory 314 corresponds to a set of instructions for performing the functions described herein. The above modules (e.g., instruction sets) do not need to be implemented as separate software programs, processes, or modules, so in various embodiments, each subset of these modules can be combined or otherwise rearranged. For example, the codec module 320 optionally does not include separate decoding and encoding modules, but uses the same set of modules to perform two sets of functions. In some embodiments, the memory 314 stores a subset of the above modules and data structures. In some embodiments, the memory 314 stores additional modules and data structures not described above, such as audio processing modules.

[0078] Although Figure 3 A server system 112 is shown according to some embodiments, but Figure 3 The present invention is intended more as a functional description of various features that may be present in one or more server systems rather than as a schematic diagram of the structures of the embodiments described herein. In practice, as will be appreciated by one of ordinary skill in the art, items shown separately may be combined and some items may be separated. For example, Figure 3 Some items shown individually in the figure can be implemented on a single server, and a single item can be implemented by one or more servers. The actual number of servers used to implement the server system 112 and how the features are distributed among the servers will vary from implementation to implementation and, optionally, will depend in part on the amount of data traffic that the server system handles during peak usage and during normal usage periods.

[0079] Example encoding procedures and techniques are described below.

[0080] The encoding processes and techniques described below may be performed at the devices and systems described above (eg, source device 102, server system 112, and / or electronic device 120). Figures 4A to 4D An exemplary coding tree structure according to some embodiments is shown. Figure 4A As shown in the first coding tree structure (400) in , some coding methods (e.g., VP9) use a 4-way partition tree starting from the 64x64 level to the 4x4 level, with some additional restrictions for block 8x8. Figure 4A , the partitioning designated as R can be called recursive because the same partitioning tree is repeated at a lower scale until the lowest 4x4 level is reached.

[0081] like Figure 4B As shown in the second coding tree structure (402) in FIG. 4 , some coding methods (e.g., AV1) extend the partition tree to a 10-way structure and increase the maximum size (e.g., superblock in VP9 / AV1 terminology) to start from 128x128. The second coding tree structure includes 4:1 / 1:4 rectangular partitions that are not in the first coding tree structure. Figure 4B The partition type with 3 subpartitions in the second row of is called T-type partition. In addition to the coding block size, the coding tree depth can also be defined to indicate the partition depth starting from the root node.

[0082] For example, in order to adapt to different local features, such as in HEVC, the coding tree unit (CTU) can be divided into coding units (CU) using a quadtree structure represented as a coding tree. In some embodiments, the decision as to whether to encode the picture area using inter-frame (temporal) prediction or intra-frame (spatial) prediction is made at the CU level. Depending on the PU partition type, each CU can be further divided into one, two, or four prediction units (PUs). Within a PU, the same prediction process is used, and relevant information is sent to the decoder on a PU-by-PU basis. After obtaining the residual block by applying the prediction process based on the PU partition type, the CU can be divided into transform units (TUs) according to another quadtree structure similar to the coding tree of the CU.

[0083] For example, a quadtree with nested multi-type trees using binary and ternary segmentation structures in VVC can replace the concept of multiple partition unit types. For example, it removes the concept of CU, PU, ​​and TU separation unless the size of the CU is too large for the maximum transform length as required, and supports more flexible CU partition shapes. In the coding tree structure, the CU can be square or rectangular. The CTU is first divided by a quadtree (also called a quadtree) structure. The leaf nodes of the quadtree can be further divided by a multi-type tree structure. Figure 4CAs shown in the third coding tree structure (404) in FIG, the multi-type tree structure includes four types of partitions. The leaf nodes of the multi-type tree are called CUs, and this partitioning is used for prediction and transform processing without any further partitioning unless the CU is too large for the maximum transform length. This means that in most cases, CUs, PUs, and TUs have the same block size in a quadtree with a nested multi-type tree coding block structure. Figure 4D An example of block partitioning of a CTU (406) is shown, showing an exemplary quadtree with a nested multi-type tree coding block structure.

[0084] For directional intra prediction, some methods support 8 directional modes corresponding to angles from 45 degrees to 207 degrees. In order to exploit more kinds of spatial redundancy in directional textures, the directional intra modes can be extended to a set of angles with finer granularity. For example, 8 angles can be represented as nominal angles. Figure 5A As shown, there are 8 nominal angles, named as: vertical prediction (V_PRED), horizontal prediction (H_PRED), 45 degree prediction (D45_PRED), 135 degree prediction (D135_PRED), 113 degree prediction (D113_PRED), 157 degree prediction (D157_PRED), 203 degree prediction (D203_PRED) and 67 degree prediction (D67_PRED). Figure 5A It is also shown that for each nominal angle, there are 7 finer angles possible, for a total of 56 directional angles. The predicted angle can be described by the nominal intra angle plus an angle increment (e.g., -3 to 3 times a step size of 3 degrees). In order to implement the directional prediction modes in a generic way, the 56 directional intra prediction modes can be implemented with a unified directional predictor that projects each pixel to a reference sub-pixel position and interpolates the reference pixel through a 2-tap bilinear filter. This directional intra prediction is sometimes also called unidirectional intra prediction.

[0085] In some methods, a lookup table is used to map each intra prediction angle to a horizontal offset and a vertical offset between each pixel in the current block and a reference sample. The offset in the lookup table can be an integer value of the tangent of each intra prediction angle multiplied by 64. For example, 64 is equal to tangent (45°) multiplied by 64, where tangent (45°) is equal to 1. For example, for a 45-degree intra prediction angle, the associated offset is 64, and the horizontal offset between each pixel in the current block and the reference pixel increases by 1 pixel as the row number of the pixel increases by 1.

[0086] In some methods, there are also non-directional smooth intra prediction modes, such as DC mode, PAETH mode, SMOOTH-B (sometimes expressed as SMOOTH), SMOOTH-V and SMOOTH-H modes. For DC prediction, the average of the left adjacent samples and the upper adjacent samples can be used as the predictor of the block to be predicted. For PAETH prediction, the upper reference sample, the left reference sample and the upper left reference sample can be obtained, and the value closest to the given value (e.g., upper + left - upper left) is set as the predictor of the pixel to be predicted. Figure 5B The positions of the upper sample 514, the left sample 518, and the upper left sample 516 of the current pixel (or small block or sub-block) in the current block are shown. For SMOOTH-B, SMOOTH-V, and SMOOTH-H modes, the block can be predicted using quadratic interpolation in the vertical or horizontal direction or an average of the two directions.

[0087] In order to obtain attenuated spatial correlation with reference on edges, filter intra modes can be used for luma blocks. In some methods, five filter intra modes are defined, each represented by a set of eight 7-tap filters that reflect the correlation between a pixel in a 4x2 block (subblock) and its 7 adjacent neighbors. In this way, the weighting factors of the 7-tap filters are position-dependent. For example, Figure 5C As shown in Figure 1, an 8×8 block can be divided into 8 4×2 small blocks. Figure 5C In the example, these blocks are represented by B0, B1, B2, B3, B4, B5, B6, and B7. For each block, 7 neighbors (represented by R0-R7) are used to predict the pixels in the current block. For block B0, all neighbors have been reconstructed. But for other blocks, some neighbors may not be reconstructed. In this case, the predicted values ​​of the nearest neighboring pixels can be used as a reference. For example, all neighbors of block B7 may not be reconstructed, so the predicted samples of neighbors (e.g., B5 and B6) can be used instead.

[0088] In some methods, SMOOTH-V and SMOOTH-H modes generate prediction values ​​by using interpolation in the vertical and horizontal directions, respectively, while SMOOTH-B mode generates prediction values ​​using the average of the interpolation results in both directions. The adjacent reconstructed sample values ​​of top, left, top right, and bottom left can be represented as T, L, TR, and BL, respectively. The predicted sample values ​​in the three modes can be calculated as follows.

[0089] P H (x, y)=w(x)×L+(1-w(x))×TR

[0090] Formula 1 - SMOOTH-H Mode

[0091] PV (x, y)=w(y)×T+(1-w(y))×BL

[0092] Formula 2-SMOOTH-mode

[0093] P(x, y) = (P H (x,y)+P V (x,y)) / 2

[0094] Formula 3 - SMOOTH-B Mode

[0095] Where x represents the horizontal coordinate of the prediction sample in the prediction block, y represents the vertical coordinate of the prediction sample in the prediction block, w(x) and w(y) are weighted values ​​based on the coordinate values, P H , P V and P represent the prediction blocks of SMOOTH-H, SMOOTH-V and SMOOTH-B modes respectively. Figure 5D The prediction process of SMOOTH-V and SMOOTH-H modes is shown, where w(y) and w(x) are nonlinear weighted values ​​derived based on the vertical coordinates and horizontal coordinates in SMOOTH-V and SMOOTH-H modes, respectively. The weights can be stored in a lookup table.

[0096] In some systems, smooth prediction modes (e.g., the SMOOTH-H, SMOOTH-V, and SMOOTH-B modes above) do not take into account the uneven distribution of available reference samples (e.g., only upper and left reference samples are available) or the nonlinear weighted distribution of available reference samples, and therefore, the prediction accuracy of these smooth prediction modes may be reduced compared to modes that give larger weights to upper and left reference samples. The systems and methods described below can improve prediction accuracy compared to the smooth prediction modes shown in formulas 1-3 above. The methods and processes described below can be used alone or in combination in any order. In the following description, if a mode generates prediction samples according to a given prediction direction, the mode is called a directional mode.

[0097] As used herein, Figure 5E As shown in the left part 528 in FIG. 1 , the left reference sample refers to a reference sample whose vertical coordinate value is within the minimum vertical coordinate value and the maximum vertical coordinate value of the current block (eg, the current block 522). Figure 5E As shown in the upper part 524 of FIG. 1 , the upper reference sample refers to a reference sample whose horizontal coordinate value is within the minimum horizontal coordinate value and the maximum horizontal coordinate value of the current block. Figure 5E As shown in the lower left portion 530, the lower left reference sample refers to a reference sample whose vertical coordinate value is greater than the maximum vertical coordinate value of the current block. Figure 5EAs shown in the upper right portion 526, the upper right reference sample refers to a reference sample whose horizontal coordinate value is greater than the maximum horizontal coordinate value of the current block. Figure 5E As shown in the upper left part 532 in FIG. 1 , the upper left reference sample refers to a reference sample whose horizontal coordinate value is smaller than the minimum horizontal coordinate value of the current block and whose vertical coordinate value is smaller than the minimum vertical coordinate value of the current block.

[0098] Fig. 6A 6 is a flow chart illustrating a video encoding method 600 according to some embodiments. The method 600 may be performed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having a control circuit and a memory storing instructions executed by the control circuit. In some embodiments, the method 600 is performed by executing instructions stored in a memory (e.g., memory 314) of the computing system.

[0099] The system receives (602) video data including a plurality of blocks (including a first block), wherein the first block is to be encoded in a smoothing mode (e.g., SMOOTH-B, SMOOTH-H, or SMOOTH-V). The system identifies (604) a reference sample set for the first block. The system derives (606) a first prediction value for the first block. The system derives (608) a refined first prediction value for the first block using a weighted sum of a first reference sample of the reference sample set and the first prediction value. The system encodes (610) the first block based on the refined first prediction value. For example, the system reconstructs the first block based on the refined first prediction value, evaluates a smoothing mode based on the reconstructed first block, and selects a smoothing mode for encoding the first block based on the evaluation (e.g., a smoothing mode with minimal error).

[0100] Figure 6B 6 is a flow chart illustrating a video decoding method 650 according to some embodiments. The method 650 may be performed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having a control circuit and a memory storing instructions executed by the control circuit. In some embodiments, the method 650 is performed by executing instructions stored in a memory (e.g., memory 314) of the computing system.

[0101] The system receives (652) video data (652) from a video bitstream including a plurality of blocks including a first block, wherein the first block is encoded in a smoothing mode (e.g., SMOOTH-B, SMOOTH-H, or SMOOTH-V). The system identifies (654) a reference sample set for the first block. The system derives (656) a first prediction value for the first block. The system derives (658) a refined first prediction value for the first block using a weighted sum of a first reference sample of the reference sample set and the first prediction value. The system decodes (660) the first block based on the refined first prediction value.

[0102] Although Fig. 6A and 6B The various logical stages are shown in a particular order, but stages that are not dependent on order may be reordered, and other stages may be combined or decomposed. Some reordering or other groupings not specifically mentioned will be apparent to one of ordinary skill in the art, and thus the ordering and groupings given herein are not exhaustive. Furthermore, it should be appreciated that the stages may be implemented in hardware, firmware, software, or any combination thereof.

[0103] In some embodiments, in order to use spatially adjacent reference samples to predict samples in the current block, a horizontal prediction value P is first obtained. H , then use the left reference sample (e.g., the reference sample with the same vertical coordinate) and P H The weighted sum of the refined level prediction value P H '. In some embodiments, the refined horizontal prediction value P H ' is used for current sample prediction (eg, corresponding to SMOOTH-H mode). In some embodiments, the vertical prediction value P is obtained V , and then use the upper reference sample (e.g., the reference sample with the same horizontal coordinate) and P V The weighted sum of the refined level prediction value P V '. In some embodiments, the refined vertical prediction value P V ' for the current sample prediction (eg, corresponding to SMOOTH-V mode). In some embodiments, the combined (final) prediction value P is taken as P H ' and P V In some embodiments, the combined prediction value P is used for current sample prediction (eg, corresponding to SMOOTH-B mode).

[0104] In some embodiments, the horizontal prediction value P is derived using a weighted sum of the left reference sample and one or more above (and / or above-right) reference samples. H For example, the weighted sum may be based on bilinear interpolation filtering. In some embodiments, the vertical prediction value P is derived using a weighted sum of an upper reference sample and one or more left (and / or lower left) reference samples. V For example, the weighted sum may be based on bilinear interpolation filtering. In some embodiments, when generating P H and / or P V Before generating the value, the left reference sample and the upper reference sample are filtered. H and P V Before the values ​​are calculated, the left reference sample and the upper reference sample may be filtered using a smoothing filter (eg, a Gaussian filter or a bilateral filter), wherein coefficients of the smoothing filter are non-negative integers.

[0105] In some embodiments, the refined level prediction value P H ' is derived using Equation 4 below.

[0106] P′ H =(w L ·L+(Nw L )·P H +r) / N

[0107] Formula 4 - Refined Horizontal Prediction Value

[0108] Where L is the left sample or the lower left sample, w L is derived using the horizontal coordinate value of the current sample, N is a predefined value (e.g., an integer of a power of 2, such as 2, 4, 8, 16, 32, 64, 128, 256, 512, or 1024), and r is a rounding offset (e.g., equal to 0.5·N). In some embodiments, w L is K>>((x<<1)>>s), where K is a predefined value (e.g., 16, 32, or 64), x is the horizontal coordinate of the current sample to be predicted, s is a scaling factor based on the block size (e.g., s=(log2(W)+log2(H)+2)>>2), and W and H are the width and height of the block, respectively.

[0109] In some embodiments, the refined vertical prediction value P V ' is derived using Equation 5 below.

[0110] P V ′=(w T ·T+(Nw T )·P V +r) / N

[0111] Formula 5 - Refined vertical prediction value

[0112] Where T is the upper sample or upper right sample, w T is derived using the vertical coordinate value of the current sample. In some embodiments, w T It is K>>((y<<1)>>s), where y is the vertical coordinate of the current sample to be predicted.

[0113] In some embodiments, the combined (final) prediction value P is obtained as P H ' and P V '. In some embodiments, P is P H ' and P V ', where the weights depend on the width and height of the block.

[0114] In some embodiments, the new modes discussed above are used with the SMOOTH, SMOOTH-V, and SMOOTH-H modes of AV1 (e.g., corresponding to equations 1-3). In some embodiments, the new modes discussed above replace the SMOOTH, SMOOTH-V, and SMOOTH-H modes of AV1. In some embodiments, the new modes discussed above are signaled as additional modes. For example, the mode can be a new mode in the intra-mode list. In another example, the mode can be signaled in the SMOOTH-B, SMOOTH-V, and SMOOTH-H modes. For example, a smooth mode index can be signaled to indicate whether to use a new mode or a pre-existing mode.

[0115] In some embodiments, when calculating the predicted samples for the new SMOOTH-B mode (or hybrid mode), the division (and / or right shift) operation is moved to the last derivation step. For example, the SMOOTH-B mode is calculated as shown in the following formula 6-8, where L is the left sample or the lower left sample, and T is the upper sample or the upper right sample.

[0116] P′ H =(w L ·L+(Nw L )·P H )

[0117] Formula 6 - Prediction of refinement level without division

[0118] P V ′ =(w T ·T+(Nw T )·P V )

[0119] Formula 7 - Refined vertical prediction value without division

[0120] P=(P′ H +P V ′+r) / (2·N)

[0121] Formula 8-Combined prediction value

[0122] In some embodiments, P is calculated H and P V The division (and / or right shift) operation when is also moved to the last derivation step of the combined (final) predictor, as shown in Equations 9-13 below. For example, during internal calculations, a common value may be multiplied so that each predictor has the same divisor (or shift value).

[0123] P V =(T·(H-1-y)+BL·(y+1))·W

[0124] Formula 9-Vertical prediction value

[0125] P V ′=(w T ·W·H·T+(Nw T )·P V )

[0126] Formula 10 - Refined vertical prediction value without division

[0127] P H =(L·(W-1-x)+TR·(x+1))·H

[0128] Formula 11 - Horizontal Prediction Value

[0129] P′ H =(w L ·W·H·L+(Nw L )·P H )

[0130] Formula 12 - Prediction of refinement level without division

[0131] P=(P′ H +P V ′+r) / (2·N·W·H)

[0132] Formula 13-Combined prediction value

[0133] In Formula 9-13, r is a rounding offset (e.g., equal to N·W·H). In Formula 9-13, multiplication of W or H can be a left shift of log2(W) or log2(H), and division by (2·N·W·H) can be a right shift of (1+log2(W)+log2(H)+log2(N)).

[0134] In some embodiments, P is calculated H , P H ',P V , and / or P V ', a first right shift value is performed at the end of the corresponding calculation, and a second right shift operation is performed when calculating P (e.g., compensating for the first right shift), as shown in the following formula 14-18, where BL is the lower left reference sample and TR is the upper right reference sample.

[0135] P V =((T·(H-1-y)+BL·(y+1))·W+r V1 )>>s V1

[0136] Formula 14 - Vertical prediction value with offset

[0137] P V′=((w T ·W·H·T+(Nw T )·P V )+r V2 )>>s V2

[0138] Formula 15 - Refined vertical prediction value with offset

[0139] P H =((L·(W-1-x)+TR·(x+1))·H+r H1 )>>s H1

[0140] Equation 16 - Horizontal Prediction with Bias

[0141] P′ H =((w L ·W·H·L+(Nw L )·P H )+r H2 )>>s H2

[0142] Formula 17 - Refinement level prediction value with offset

[0143]

[0144] Formula 18 - Combined prediction value with offset

[0145] In some embodiments, (for example, if s V1 +s V2 Equal to s H1 +s H2 )Through P H and P V The final prediction P is obtained by taking the weighted average of , as shown in the following formula 19:

[0146]

[0147] Formula 19-Combined prediction value

[0148] Where r is the rounding offset, an example value is

[0149] (A1) In one aspect, some embodiments include a method of video encoding (e.g., method 600). In some embodiments, the method is performed on a computing system (e.g., server system 112) having a memory and a control circuit. In some embodiments, the method is performed at a codec module (e.g., codec module 320). In some embodiments, the method is performed at a source encoding component (e.g., source encoder 202), an encoding engine (e.g., encoding engine 212), and / or an entropy encoder (e.g., entropy encoder 214). The method includes: (i) receiving video data including a plurality of blocks, the plurality of blocks including a first block, wherein the first block is to be encoded in a smooth mode; (ii) identifying a reference sample set for the first block; (iii) deriving a first prediction value for the first block; (iv) deriving a refined first prediction value for the first block using a weighted sum of a first reference sample of the reference sample set and the first prediction value; and (v) encoding the first block based on the refined first prediction value.

[0150] (A2) In some embodiments of A1, the first prediction value is a horizontal prediction value and the first reference sample is a left reference sample. In some embodiments, the weighted sum of the left reference sample and the horizontal prediction value uses a weight of the left reference sample based on the horizontal coordinate of the sample of the first block to be encoded.

[0151] (A3) In some embodiments of A1, the first prediction value is a vertical prediction value and the first reference sample is an upper reference sample. In some embodiments, the weighted sum of the upper reference sample and the vertical prediction value uses a weight of the upper reference sample based on the vertical coordinate of the sample of the first block to be encoded.

[0152] (A4) In some embodiments of any one of A1-A3, the method further includes: (i) deriving a second prediction value for the first block; (ii) deriving a refined second prediction value for the first block using a weighted sum of a second reference sample of the reference sample set and the second prediction value; and (iii) deriving a combined prediction value using a weighted sum of the refined first prediction value and the refined second prediction value, wherein the first block is decoded using the combined prediction value. In some embodiments, deriving the combined prediction value using the weighted sum includes using a weight based on a block width and a block height of the first block.

[0153] (A5) In some embodiments of any of A1-A4, a weighted sum of a left reference sample and an above reference sample (eg, using Equation 9 or 11) is used to derive a first prediction value for the first block.

[0154] (A6) In some embodiments of any one of A1-A5, the method further includes filtering the set of reference samples to identify the first reference sample.

[0155] (A7) In some embodiments of any one of A1-A6, the method further comprises sending the encoded first block via a video bitstream.

[0156] (B1) On the other hand, some embodiments include a method of video decoding (e.g., method 650). In some embodiments, the method is performed on a computing system (e.g., server system 112) having a memory and a control circuit. In some embodiments, the method is performed at a codec module (e.g., codec module 320). In some embodiments, the method is performed at a parser (e.g., parser 254), a motion prediction component (e.g., motion compensation prediction unit 260), and / or an intra-frame prediction component (e.g., intra-frame prediction unit 262). The method includes: (i) receiving video data including a plurality of blocks (e.g., an encoded video sequence) from a video bitstream, the plurality of blocks including a first block; wherein the first block is encoded in a smooth mode; (ii) identifying a reference sample set for the first block; (iii) deriving a first prediction value for the first block; (iv) deriving a refined first prediction value for the first block using a weighted sum of a first reference sample of the reference sample set and the first prediction value; and (v) decoding the first block based on the refined first prediction value. For example, the smoothness mode is SMOOTH-H mode, SMOOTH-V mode or combined (SMOOTH-B) mode.In some embodiments, a first syntax element in the video bitstream is used to signal the smoothness mode.

[0157] (B2) In some embodiments of B1, the first prediction value is a horizontal prediction value, and the first reference sample is a left reference sample. For example, in order to use spatially adjacent reference samples to predict samples in the current block, the horizontal prediction value P is first obtained. H , then use the left reference sample and P H The weighted sum of is used to obtain the final prediction value P. In some embodiments, one of Formulas 4, 6, 12 or 17 is used to obtain the refined first prediction value.

[0158] (B3) In some embodiments of B2, the weighted sum of the left reference sample and the horizontal prediction value uses the weight of the left reference sample based on the horizontal coordinate of the sample of the first block to be predicted. For example, w L is K>>((x<<1)>>s), where K is a predefined value, x is the horizontal coordinate of the current sample to be predicted, and s is a scaling factor based on the block size. For example, s can be equal to (log2(W)+log2(H)+2)>>2, where W and H are the width and height of the block, respectively.

[0159] (B4) In some embodiments of B1, the first prediction value is a vertical prediction value, and the first reference sample is an upper reference sample. For example, in order to use spatially adjacent reference samples to predict samples in the current block, the vertical prediction value P is first obtained. V , then use the reference sample and P V The weighted sum of is used to obtain the final prediction value P. In some embodiments, one of Formulas 5, 7, 10, or 15 is used to refine the first prediction value.

[0160] (B5) In some embodiments of B4, the weighted sum of the upper reference sample and the vertical prediction value uses the weight of the upper reference sample based on the vertical coordinate of the sample of the first block to be predicted. For example, w T is K>>((y<<1)>>s), where K is a predefined value, y is the vertical coordinate of the current sample to be predicted, and s is a scaling factor based on the block size. For example, s can be equal to (log2(W)+log2(H)+2)>>2, where W and H are the width and height of the block, respectively.

[0161] (B6) In some embodiments of any one of B1-B5, the method further comprises: (i) deriving a second prediction value for the first block; (ii) deriving a refined second prediction value for the first block using a weighted sum of a second reference sample of the reference sample set and the second prediction value; and (iii) deriving a combined prediction value using a weighted sum of the refined first prediction value and the refined second prediction value, wherein the first block is decoded using the combined prediction value. For example, in order to predict samples in the current block using spatially adjacent reference samples, a horizontal prediction value P is first derived. H , then use the left reference sample (e.g., the reference sample with the same vertical coordinate) and P H The weighted sum of the refined level prediction value P H '. In this example, the vertical prediction value P is obtained V , and then use the upper reference sample (e.g., the reference sample with the same horizontal coordinate) and P V The weighted sum of the refined level prediction value P V '. Then, in this example, the final predicted value P is P H ' and P V 'The weighted sum.

[0162] (B7) In some embodiments of B6, using the weighted sum to derive the combined prediction value includes using a weight based on the block width and block height of the first block. For example, the combined prediction value P is P H ' and P V ', where the weights depend on the width and height of the block. In some embodiments, the combined prediction value P is obtained as P H ' and PV 'The average value.

[0163] (B8) In some embodiments of any of B1-B7, a weighted sum of a left reference sample and an upper reference sample (e.g., using Equation 9 or 11) is used to derive a first prediction value for the first block. For example, a weighted sum of a left reference sample and one (or more) upper (and / or upper-right) reference sample is used to derive a horizontal prediction value P H In one example, the weighted sum uses bilinear interpolation filtering. As another example, the vertical prediction value P is derived using a weighted sum of an upper reference sample and one (or more) left (and / or lower left) reference sample. V .

[0164] (B9) In some embodiments of any one of B1-B8, the method further includes filtering the reference sample set to identify the first reference sample. H and P V In some embodiments, the left reference sample and the upper reference sample are filtered before generating the P value. H and P V Before the values ​​are calculated, the left reference sample and the upper reference sample are filtered using a smoothing filter (such as a Gaussian filter or a bilateral filter). For example, the coefficients of the smoothing filter can all be non-negative integers.

[0165] On the other hand, some embodiments include a computing system (e.g., server system 112) that includes a control circuit (e.g., control circuit 302) and a memory (e.g., memory 314) coupled to the control circuit, the memory storing one or more sets of instructions configured to be executed by the control circuit, the one or more sets of instructions including instructions for performing any of the methods described herein (e.g., A1-A7 and B1-B9 above).

[0166] On the other hand, some embodiments include a non-transitory computer-readable storage medium storing one or more sets of instructions executed by control circuitry of a computing system, the one or more sets of instructions including instructions for performing any of the methods described herein (e.g., A1-A7 and B1-B9 above).

[0167] It should be understood that although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another.

[0168] The terms used herein are only used to describe specific embodiments and are not intended to limit the claims. As used in the description of the embodiments and the appended claims, unless the context clearly indicates otherwise, the singular forms "one", "an" and "the" are also intended to include plural forms. It should also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more associated listed items. It should also be understood that when used in this specification, the terms "include" and / or "comprising" specify the presence of the features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0169] As used herein, the term "if" may be interpreted to mean "when" or "at" or "in response to determining" or "according to determining" or "in response to detecting" these indicate that the precondition is true, depending on the context. Likewise, the phrase "if it is determined that the precondition is true" or "if the precondition is true" or "when the precondition is true" may be interpreted to mean "upon determination" or "in response to determining" or "according to determining" or "upon detecting" or "in response to detecting" the precondition is true, depending on the context.

[0170] For purposes of explanation, the above description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the claims to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments were chosen and described in order to best explain the principles of operation and practical application, thereby enabling those skilled in the art to use.

Claims

1. A video decoding method, the method being executed on a computing system having a memory and one or more processors, characterized in that: The method comprises: receiving video data comprising a plurality of blocks from a video bitstream, the plurality of blocks comprising a first block, wherein the first block is encoded in one of a plurality of smooth modes; identifying a set of reference samples for the first block; Obtaining a first prediction value of the first block; deriving a refined first prediction value of the first block using a weighted sum of a first reference sample of the reference sample set and the first prediction value, wherein a first weight of the first reference sample and a second weight of the first prediction value are both non-zero; and The first block is decoded based on the refined first prediction value.

2. The method according to claim 1, characterized in that The first prediction value is a horizontal prediction value, and wherein the first reference sample is a left reference sample.

3. The method according to claim 2, characterized in that The weighted sum of the left reference sample and the horizontal prediction value uses a weight of the left reference sample, the weight being based on a horizontal coordinate of a sample of the first block to be predicted.

4. The method according to claim 1, characterized in that: The first prediction value is a vertical prediction value, and wherein the first reference sample is an upper reference sample.

5. The method according to claim 4, characterized in that The weighted sum of the upper reference sample and the vertical prediction value uses a weight of the upper reference sample, the weight being based on a vertical coordinate of a sample of the first block to be predicted.

6. The method according to claim 1, characterized in that The method further comprises: deriving a second prediction value of the first block; deriving a refined second prediction value of the first block using a weighted sum of a second reference sample of the reference sample set and the second prediction value; deriving a combined prediction value using a weighted sum of the refined first prediction value and the refined second prediction value; and The first block is decoded using the combined prediction value.

7. The method according to claim 6, characterized in that The using the weighted sum to derive the combined prediction value includes using a weight based on a block width and a block height of the first block.

8. The method according to claim 1, characterized in that The first prediction value of the first block is derived using a weighted sum of left and above reference samples.

9. The method according to claim 1, characterized in that: The method also includes filtering the set of reference samples to identify the first reference sample.

10. A computing system, characterized in that: The computing system comprises: Control circuit; Memory; and One or more sets of instructions, the one or more sets of instructions stored in the memory and configured to be executed by the control circuit, the one or more sets of instructions comprising instructions for implementing the following operations: receiving video data comprising a plurality of blocks from a video bitstream, the plurality of blocks comprising a first block, wherein the first block is encoded in one of a plurality of smooth modes; identifying a set of reference samples for the first block; Obtaining a first prediction value of the first block; deriving a refined first prediction value of the first block using a weighted sum of a first reference sample of the reference sample set and the first prediction value, wherein a first weight of the first reference sample and a second weight of the first prediction value are both non-zero; and The first block is decoded based on the refined first prediction value.

11. The computer system according to claim 10, characterized in that: The first prediction value is a horizontal prediction value, wherein the first reference sample is a left reference sample.

12. The computing system according to claim 10, characterized in that: The first prediction value is a vertical prediction value, and wherein the first reference sample is an upper reference sample.

13. The computing system according to claim 10, characterized in that: The one or more sets of instructions also include instructions for implementing the following operations: deriving a second prediction value of the first block; deriving a refined second prediction value of the first block using a weighted sum of a second reference sample of the reference sample set and the second prediction value; deriving a combined prediction value using a weighted sum of the refined first prediction value and the refined second prediction value; and Wherein, the first block is decoded using the combined prediction value.

14. The computing system according to claim 13, characterized in that: The using the weighted sum to derive the combined prediction value includes using a weight based on a block width and a block height of the first block.

15. The computing system according to claim 10, characterized in that: The first prediction value of the first block is derived using a weighted sum of left and above reference samples.

16. A non-transitory computer-readable storage medium storing one or more sets of instructions, characterized in that: The one or more sets of instructions are configured to be executed by a computing device having control circuitry and memory, and the one or more sets of instructions include instructions for implementing the following operations: receiving video data comprising a plurality of blocks from a video bitstream, the plurality of blocks comprising a first block, wherein the first block is encoded in one of a plurality of smooth modes; identifying a set of reference samples for the first block; Obtaining a first prediction value of the first block; deriving a refined first prediction value of the first block using a weighted sum of a first reference sample of the reference sample set and the first prediction value, wherein a first weight of the first reference sample and a second weight of the first prediction value are both non-zero; and The first block is decoded based on the refined first prediction value.

17. The non-transitory computer-readable storage medium of claim 16, wherein: The first prediction value is a horizontal prediction value, and wherein the first reference sample is a left reference sample.

18. The non-transitory computer-readable storage medium of claim 16, wherein: The first prediction value is a vertical prediction value, and wherein the first reference sample is an upper reference sample.

19. The non-transitory computer-readable storage medium of claim 16, wherein: The one or more sets of instructions also include instructions for implementing the following operations: deriving a second prediction value of the first block; deriving a refined second prediction value of the first block using a weighted sum of a second reference sample of the reference sample set and the second prediction value; deriving a combined prediction value using a weighted sum of the refined first prediction value and the refined second prediction value; and Wherein, the first block is decoded using the combined prediction value.

20. The non-transitory computer-readable storage medium of claim 16, wherein: The first prediction value of the first block is derived using a weighted sum of left and above reference samples.

Images