Bi-prediction without signaling CU-level weights

JP7584539B2Active Publication Date: 2024-11-15TENCENT AMERICA LLC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2022564807
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2022-04-21
Filing Date
2022-04-26
Publication Date
2024-11-15
Estimated Expiration
2042-04-26

AI Technical Summary

Benefits of technology

【0030】 開示された主題のさらなる特徴、性質、および様々な利点は、以下の詳細な説明および添付の図面からより明らかになるであろう。

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007584539000010
    Figure 0007584539000010
  • Figure 0007584539000011
    Figure 0007584539000011
  • Figure 0007584539000012
    Figure 0007584539000012
Patent Text Reader

Abstract

The processing circuit decodes information of the coded block in the current picture from the bitstream. This information indicates a bi-predictive mode without weight signaling. Weights associated with the bi-predictive mode are not signaled in the bitstream. Furthermore, the processing circuit determines a first motion vector associated with the first reference picture and a second motion vector associated with the second reference picture, and determines a first reference template in the first reference picture based on the current template and the first motion vector of the coded block, and a second reference template in the second reference picture based on the current template and the second motion vector. The processing circuit also calculates weights to be used in the bi-predictive mode based on the first reference template, the second reference template, and the current template, and reconstructs the coded block using bi-prediction with the calculated weights.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims benefit of priority to U.S. Provisional Application No. 63 / 179,928, entitled "BI-PREDICTION WITH CU-LEVEL WEIGHTS WITHOUT SIGNALING," filed April 26, 2021, which claims benefit of priority to U.S. Provisional Application No. 17 / 726,042, entitled "BI-PREDICTION WITH CU-LEVEL WEIGHTS," filed April 21, 2022. The disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002] This disclosure generally describes embodiments relating to video encoding. [Background technology]

[0003] The description of the background art provided herein is intended to generally present the context of the present disclosure. To the extent that it is described in this background art section, aspects of the inventors' work and description that may not be admitted as prior art at the time of filing are not admitted expressly or impliedly as prior art to the present disclosure.

[0004] Uncompressed digital video may include a sequence of pictures, each with spatial dimensions of, for example, 1920x1080 luma samples and associated chroma samples. The sequence of pictures may have a fixed or variable (also informally known as frame rate) picture rate of, for example, 60 pictures per second or 60 Hz. Uncompressed video has specific bitrate requirements. For example, 1080p60 4:2:0 video (1920x1080 luma sample resolution at 60 Hz frame rate) at 8 bits per sample requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 Gbytes of storage space.

[0005] One goal of video encoding and decoding may be to reduce redundancy in the input video signal through compression. Compression may help reduce the aforementioned bandwidth and / or storage space requirements, in some cases by more than two orders of magnitude. Both lossless and lossy compression may be employed, as well as combinations of these. Lossless compression refers to techniques where an exact copy of the original signal may be reconstructed from the compressed original signal. With lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal useful for the intended application. For video, lossy compression is widely adopted. The amount of acceptable distortion depends on the application, e.g., a user of a particular consumer streaming application may tolerate higher distortion than a user of a television distribution application. The achievable compression ratio may reflect that a higher acceptable / tolerable distortion may result in a higher compression ratio.

[0006] Video encoders and decoders can utilize techniques from a number of broad categories, including, for example, motion compensation, transform, quantization, and entropy coding.

[0007] Video codec techniques can include a technique called intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into blocks of samples. When all blocks of samples are coded in intra mode, the picture may be an intra picture. Intra pictures, and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state, and therefore can be used as the first picture in a coded video bitstream and video session, or as a still image. Samples of an intra block may undergo a transform, and the transform coefficients may be quantized before entropy coding. Intra prediction may be a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value and the smaller the AC coefficients after the transform, the fewer bits are required at a given quantization step size to represent the block after entropy coding.

[0008] Conventional intra-coding, for example as known from MPEG-2 generation encoding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to predict from surrounding sample data and / or metadata obtained during encoding and / or decoding of a data block that is, for example, spatially adjacent and preceding in decoding order. Such techniques are hereafter referred to as "intra-prediction" techniques. It should be noted that in at least some cases, intra-prediction uses only reference data from the current picture being reconstructed, and not from a reference picture.

[0009] Intra prediction can take many different forms. When two or more of such techniques can be used in a given video coding technique, the technique in use can be coded in intra prediction mode. In certain cases, a mode can have sub-modes and / or parameters that can be coded separately or included in the codeword of the mode. Which codeword is used for a given mode, sub-mode, and / or parameter combination can affect the coding efficiency gains via intra prediction, and so can the entropy coding technique used to convert the codeword into a bitstream.

[0010] A specific mode of intra prediction was introduced in H.264, improved in H.265, and further improved in newer coding techniques such as Joint Search Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). A predictor block can be formed using neighboring sample values ​​belonging to already available samples. The sample values ​​of the neighboring samples are copied to the predictor block according to the direction. The reference to the direction in use can be coded in the bitstream or can be predicted itself.

[0011] Referring to FIG. 1A, depicted at the bottom right is a subset of 9 predictor directions known from the 33 possible predictor directions (corresponding to the 33 angle modes of the 35 intra modes) of H.265. The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction in which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from one or more samples to the upper right at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples to the lower left of sample (101) at an angle of 22.5 degrees from the horizontal.

[0012] 1A, at the top left is shown a square block (104) of 4×4 samples (depicted by a thick dashed line). The square block (104) contains 16 samples, each labeled with "S", the sample's position in the Y dimension (e.g., row index), and the sample's position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample in the block (104) in both the Y and X dimensions. Since the block is 4×4 samples in size, S44 is at the bottom right. Also shown are reference samples that follow a similar numbering scheme. The reference samples are labeled with R, their Y position (e.g., row index), and X position (column index) relative to the block (104). In both H.264 and H.265, since the predicted samples are adjacent to the block being reconstructed, negative values ​​do not need to be used.

[0013] Intra-picture prediction can work by copying reference sample values ​​from adjacent samples according to the signaled prediction direction. For example, assume that the coded video bitstream includes signaling indicating a prediction direction that coincides with the arrow (102) for this block, i.e., the sample is predicted from one or more prediction samples located at an angle of 45 degrees from the horizontal and to the upper right. In that case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014] In certain cases, especially when the orientation is not evenly divisible by 45 degrees, the values ​​of multiple reference samples may be combined, for example by interpolation, to calculate the reference sample.

[0015] The number of possible directions is increasing as video coding techniques develop. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions at the time of disclosure. Experiments have been carried out to identify the most likely directions, and certain techniques of entropy coding are used to represent those likely directions with a small number of bits, accepting a certain penalty for less likely directions. Furthermore, the direction itself may be predicted from neighboring directions used in adjacent already decoded blocks.

[0016] FIG. 1B shows a schematic diagram (110) depicting 65 intra prediction directions with JEM to illustrate the increasing number of prediction directions over time.

[0017] The mapping of intra-prediction direction bits in a coded video bitstream to represent directions may vary from one video coding technique to another, ranging from simple direct mappings from prediction directions to intra-prediction modes, codewords, complex adaptation schemes involving most likely modes, and similar techniques. In all cases, however, there may be certain directions that are statistically less likely to occur in the video content than certain other directions. Because the goal of video compression is redundancy reduction, these less likely directions are represented with more bits than more likely directions in a well-performing video coding technique.

[0018] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. Motion compensation can be a lossy compression technique and can refer to a technique in which blocks of sample data from a previously reconstructed picture or part thereof (reference picture) are used to predict a newly reconstructed picture or part of a picture after being spatially shifted in a direction indicated by a motion vector (hereafter MV). In some cases, the reference picture can be the same as the picture currently being reconstructed. The MV can have two dimensions X and Y, or three dimensions, with the third dimension being an indication of the reference picture in use (the latter can indirectly be a temporal dimension).

[0019] In some video compression techniques, the MV applicable to a particular region of sample data can be predicted from other MVs, e.g., MVs associated with another region of sample data that is spatially adjacent to the area under reconstruction and precedes that MV in decoding order. Doing so can significantly reduce the amount of data required to encode the MV, thereby eliminating redundancy and increasing the compression ratio. For example, when encoding an input video signal derived from a camera (known as natural video), MV prediction can work effectively because there exists a statistical possibility that regions larger than the region to which a single MV is applicable move in a similar direction, and thus, in some cases, can be predicted using similar motion vectors derived from MVs of neighboring regions. As a result, the detected MV for a given region is similar or the same as the MV predicted from the surrounding MVs, and after entropy encoding, can be represented with fewer bits than would be used to encode the MV directly. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, the MV prediction itself can be lossy, e.g., due to rounding errors when computing the predictor from several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms that H.265 offers, the one described herein is a technique hereafter referred to as "spatial merging".

[0021] Referring to Figure 2, the current block (201) contains samples found by the encoder during the motion search process to be predictable from a previous block of the same size but spatially shifted. Instead of directly encoding the MV, the MV can be derived from metadata associated with one or more reference pictures, e.g., the MV can be derived from the latest (decoding order) reference picture using the MV associated with any one of the surrounding five samples represented as A0, A1 and B0, B1, B2 (202 to 206, respectively). In H.265, MV prediction can use predictors from the same reference picture that neighboring blocks use. Summary of the Invention [Means for solving the problem]

[0022] Aspects of the present disclosure provide a method and apparatus for video encoding / decoding. In some examples, the apparatus for video decoding includes a processing circuit. For example, the processing circuit decodes information of a coding block in a current picture from a bitstream. The information indicates a bi-prediction mode without weight signaling. The weight associated with the bi-prediction mode is not signaled in the bitstream. Furthermore, the processing circuit determines a first motion vector associated with a first reference picture and a second motion vector associated with a second reference picture for bi-prediction of the coding block, and determines a first reference template in the first reference picture based on a current template and the first motion vector of the coding block, and a second reference template in the second reference picture based on a current template and the second motion vector of the coding block. The processing circuit also calculates weights for use in the bi-prediction mode based on the first reference template, the second reference template, and the current template, and reconstructs the coding block using bi-prediction with the calculated weights.

[0023] According to some aspects of the disclosure, the current template includes one or more reconstructed samples adjacent to the coding block. In some examples, the current template includes at least one of one or more rows of samples above the coding block and one or more columns of samples to the left of the coding block. In some examples, the current template includes at least one of a row of samples immediately above the coding block and a column of samples immediately to the left of the coding block.

[0024] In some embodiments, the processing circuit determines weights that minimize a predetermined cost function based on the first reference template, the second reference template, and the current template. In some examples, the predetermined cost function is based on respective sample differences of the current template relative to a predicted current template predicted based on the first reference template and the second reference template using a predetermined bi-prediction model having one or more weight parameters.

[0025] In some examples, the predetermined bi-predictive model includes three weighting parameters. The processing circuit determines values ​​of the three weighting parameters in the predetermined bi-predictive model by a least squares method.

[0026] In some examples, the predetermined bi-predictive model includes two weighting parameters. The processing circuit determines values ​​of the two weighting parameters in the predetermined bi-predictive model by a least squares method.

[0027] In some examples, the predetermined bi-predictive model includes one weighting parameter. The processing circuit determines a value of the one weighting parameter in the predetermined bi-predictive model by a least squares method.

[0028] In some examples, the processing circuitry clips the weight(s) within a predetermined range.

[0029] Aspects of the present disclosure also provide a non-transitory computer-readable medium having stored thereon instructions that, when executed by a computer, cause the computer to perform a method for video decoding.

[0030] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0031] [Figure 1A] FIG. 2 is a schematic diagram of an example subset of intra-prediction modes. [Figure 1B] FIG. 2 is a diagram of an example intra-prediction direction. [Diagram 2] FIG. 2 is a schematic diagram of a current block and its surrounding spatial merge candidates in one example. [Diagram 3] 1 is a simplified schematic block diagram of a communication system (300) according to one embodiment. [Figure 4] FIG. 4 is a simplified schematic block diagram of a communication system (400) according to one embodiment. [Diagram 5] FIG. 2 is a schematic diagram of a simplified block diagram of a decoder according to one embodiment. [Figure 6] FIG. 2 is a schematic diagram of a simplified block diagram of an encoder according to one embodiment. [Figure 7] FIG. 4 is a block diagram of an encoder according to another embodiment. [Figure 8] FIG. 4 is a block diagram of a decoder according to another embodiment. [Figure 9A] FIG. 2 illustrates uni-prediction of an inter-coded coding block in one example. [Figure 9B] FIG. 11 is a diagram showing uni-prediction of an inter-coded coding block in another example. [Figure 9C] FIG. 13 is a diagram showing uni-prediction of an inter-coded coding block in yet another example. [Figure 10] FIG. 2 is a diagram illustrating the use of bi-prediction to encode a video sequence, according to some embodiments of the present disclosure. [Figure 11] 1 is a flowchart outlining a process for processing in a bi-predictive mode according to one embodiment of the present disclosure. [Figure 12] 1 is a flowchart outlining a process according to some embodiments of the present disclosure. [Figure 13] 1 is a flowchart outlining an encoding process according to one embodiment of the present disclosure. [Figure 14] 1 is a flowchart outlining a decoding process according to one embodiment of the present disclosure. [Figure 15] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] FIG. 3 illustrates a simplified block diagram of a communication system (300) according to one embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices that can communicate with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) perform unidirectional transmission of data. For example, the terminal device (310) can encode video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to another terminal device (320) via the network (350). The encoded video data can be transmitted in the form of one or more encoded video bitstreams. The terminal device (320) can receive the encoded video data from the network (350), decode the encoded video data to reconstruct the video pictures, and display the video pictures according to the reconstructed video data. Unidirectional data transmission may be common, such as in media serving applications.

[0033] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) performing bidirectional transmission of encoded video data, such as may occur during a video conference. In the case of bidirectional transmission of data, in one example, each of the terminal devices (330) and (340) can encode video data (e.g., a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) over the network (350). Each of the terminal devices (330) and (340) can also receive encoded video data transmitted by the other of the terminal devices (330) and (340), can decode the encoded video data to recover the video pictures, and can display the video pictures on an accessible display device according to the recovered video data.

[0034] In the example of FIG. 3, terminal devices (310), (320), (330), and (340) may be depicted as a server, a personal computer, and a smartphone, although the principles of the present disclosure may not be so limited. Embodiments of the present disclosure find application with laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (350) represents any number of networks that convey encoded video data between terminal devices (310), (320), (330), and (340), including, for example, wireline and / or wireless communication networks. Communications network (350) may exchange data over circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of network (350) may not be important to the operation of the present disclosure unless otherwise described herein below.

[0035] 4 shows an arrangement of a video encoder and a video decoder in a streaming environment as an example of an application of the disclosed subject matter. The disclosed subject matter may be equally applicable to other video-enabled applications including, for example, video conferencing, digital TV, and storage of compressed video on digital media including CDs, DVDs, memory sticks, and the like.

[0036] The streaming system may include a video source (401) and a capture subsystem (413) that may include, for example, a digital camera that creates, for example, a stream of uncompressed video pictures (402). In one example, the stream of video pictures (402) includes samples taken by a digital camera. The stream of video pictures (402), depicted as thick lines to emphasize the large amount of data when compared to the encoded video data (404) (or encoded video bitstream), may be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)), depicted as thin lines to emphasize the small amount of data when compared to the stream of video pictures (402), may be stored in a streaming server (405) for future use. One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to obtain copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes an input copy (407) of the encoded video data and creates an output stream (411) of video pictures that can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., a video bitstream) can be encoded according to a particular video encoding / compression standard. Examples of these standards include ITU-T Recommendation H.265. In one example, the video encoding standard under development is informally known as Versatile Video Coding (VVC). DISCLOSURE The subject matter described may be used in connection with VVC.

[0037] It should be noted that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0038] 5 shows a block diagram of a video decoder (510) according to one embodiment of the present disclosure. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used in place of the video decoder (410) of the example of FIG. 4.

[0039] The receiver (531) may receive one or more coded video sequences, in the same or another embodiment, one coded video sequence at a time, to be decoded by the video decoder (510), with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences may be received from a channel (501), which may be a hardware / software link to a storage device that stores the coded video data. The receiver (531) may receive the coded video data along with other data, e.g., coded audio data and / or auxiliary data streams, that may be forwarded to a respective using entity (not shown). The receiver (531) may separate the coded video sequences from the other data. To combat network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter "parser (520)"). In certain applications, the buffer memory (515) is part of the video decoder (510). In other applications, the buffer memory may be external to the video decoder (510) (not shown). In still other applications, there may be a buffer memory (not shown) external to the video decoder (510), for example to combat network jitter, plus another buffer memory (515) internal to the video decoder (510), for example to handle playout timing. If the receiver (531) is receiving data from a storage / forwarding device with sufficient bandwidth and controllability or from an isochronous network, the buffer memory (515) may not be needed or may be small. For use with best-effort packet networks such as the Internet, a buffer memory (515) may be needed, may be relatively large, may be advantageously adaptively sized, and may be implemented at least in part in an operating system or similar element (not shown) external to the video decoder (510).

[0040] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the coded video sequence. These categories of symbols include information used to manage the operation of the video decoder (510) and potentially information for controlling a rendering device, such as a rendering device (512) (e.g., a display screen) that is not an integral part of the electronic device (530) but may be coupled to the electronic device (530) as shown in FIG. 5. The control information for the rendering device may be in the form of a supplemental enhancement information (SEI message) or a video usability information (VUI) parameter set fragment (not shown). The parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow a variety of principles including variable length coding, Huffman coding, arithmetic coding, with or without context-dependent, etc. The parser (520) may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser (520) may also extract information from the coded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0041] The parser (520) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to produce symbols (521).

[0042] The reconstruction of the symbols (521) can involve a number of different units, depending on the type of encoded video picture or part thereof (inter-picture and intra-picture, inter-block and intra-block, etc.), and other factors. Which units are involved and how can be controlled by subgroup control information parsed from the encoded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following units is not depicted for the sake of clarity.

[0043] Beyond the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In an actual implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate.

[0044] The first unit is a scalar / inverse transform unit (551). The scalar / inverse transform unit (551) receives quantized transform coefficients as well as control information including which transform to use, block size, quantization coefficients, quantization scaling matrices, etc. as symbols (521) from the parser (520). The scalar / inverse transform unit (551) can output blocks containing sample values ​​that can be input to an aggregator (555).

[0045] In some cases, the output samples of the scaler / inverse transform (551) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from a current picture buffer (558). The current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (555) adds, possibly on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551).

[0046] In other cases, the output samples of the scalar / inverse transform unit (551) may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion compensated prediction unit (553) may access a reference picture memory (557) to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols (521) associated with the block, these samples may be added to the output of the scalar / inverse transform unit (551) by an aggregator (555) to generate output sample information (in this case referred to as residual samples or residual signals). The addresses in the reference picture memory (557) from which the motion compensated prediction unit (553) fetches the prediction samples may be controlled by motion vectors available to the motion compensated prediction unit (553) in the form of symbols (521), which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory (557) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.

[0047] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). Video compression techniques may include in-loop filter techniques that are controlled by parameters contained in the coded video sequence (also called coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), but may also be responsive to meta-information obtained during decoding of a previous portion (in decoding order) of the coded picture or coded video sequence, or to previously reconstructed and loop filtered sample values.

[0048] The output of the loop filter unit (556) may be a sample stream that can be stored in a reference picture memory (557) for use in future inter-picture prediction as well as output to a rendering device (512).

[0049] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future predictions. For example, once a coded picture corresponding to a picture of interest is fully reconstructed and the coded picture is identified (e.g., by the parser (520)) as a reference picture, the current picture buffer (558) can become part of the reference picture memory (557), and a new current picture buffer can be reallocated before starting the reconstruction of the next coded picture.

[0050] The video decoder (510) may perform decoding operations according to a given video compression technique in a standard such as ITU-T Rec. H.265. The encoded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that the encoded video sequence adheres to both the syntax of the video compression technique or standard and the profile documented in the video compression technique. Specifically, the profile may select a particular tool from all tools available in the video compression technique or standard as the only tool available for use under that profile. Also, what is required for compliance may be that the complexity of the encoded video sequence is within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further limited in some cases by a hypothetical reference decoder (HRD) specification and HRD buffer management metadata signaled in the encoded video sequence.

[0051] In one embodiment, the receiver (531) can receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0052] 6 shows a block diagram of a video encoder (603) according to one embodiment of the present disclosure. The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used in place of the video encoder (403) of the example of FIG. 4.

[0053] The video encoder (603) may receive video samples from a video source (601) (which in the example of FIG. 6 is not part of the electronic device (620)), which may capture the video image(s) that are encoded by the video encoder (603). In another example, the video source (601) is part of the electronic device (620).

[0054] The video source (601) may provide a source video sequence to be encoded by the video encoder (603) in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (601) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each of which may contain one or more samples, depending on the sampling structure, color space, etc., in use. Those skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0055] According to one embodiment, the video encoder (603) may encode and compress pictures of a source video sequence into an encoded video sequence (643) in real time or under any other time constraint required by the application. Enforcing an appropriate encoding rate is one function of the controller (650). In some embodiments, the controller (650) controls and is operatively coupled to other functional units described below. For clarity, coupling is not shown. Parameters set by the controller (650) may include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other appropriate functions associated with the video encoder (603) optimized for a particular system design.

[0056] In some embodiments, the video encoder (603) is configured to operate in an encoding loop. As an oversimplified explanation, in one example, the encoding loop may include a source coder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to that which the (remote) decoder also creates (since any compression between the symbols and the encoded video bitstream in the video compression techniques contemplated in the disclosed subject matter is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream results in bit-exact results regardless of the location of the decoder (local or remote), the contents in the reference picture memory (634) are also bit-exact between the local and remote encoders. In other words, the prediction part of the encoder "sees" exactly the same sample values ​​as the reference picture samples as the decoder "sees" when using the prediction during decoding. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, e.g., due to channel errors) is also used in several related technologies.

[0057] The operation of the "local" decoder (633) may be the same as the operation of a "remote" decoder, such as the video decoder (510), already described in detail above in conjunction with Figure 5. However, and with brief reference again to Figure 5, because symbols are available and the encoding / decoding of symbols into an encoded video sequence by the entropy coder (645) and parser (520) may be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory (515), and the parser (520), may not be fully implemented in the local decoder (633).

[0058] An observation that can be made at this point is that decoder techniques other than analysis / entropy decoding present in a decoder necessarily must also be present in the corresponding encoder, in substantially identical functional form. For this reason, the disclosed subject matter focuses on the operation of the decoder. A description of the encoder techniques can be omitted, since they are the inverse of the decoder techniques described generically. Only in certain areas is a more detailed description necessary, which is provided below.

[0059] During operation, in some examples, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (632) codes differences between pixel blocks of the input picture and pixel blocks of the reference picture(s) that may be selected as predictive reference(s) to the input picture.

[0060] The local video decoder (633) may decode the encoded video data of pictures that may be designated as reference pictures based on the symbols created by the source coder (630). The operation of the encoding engine (632) may advantageously be a lossy process. When the encoded video data may be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may be a copy of the source video sequence, usually with some errors. The local video decoder (633) may replicate the decoding process that may be performed by the video decoder on the reference pictures, such that the reconstructed reference pictures are stored in the reference picture cache (634). In this way, the video encoder (603) may locally store copies of reconstructed reference pictures that have common content as reconstructed reference pictures obtained by a far-end video decoder (without transmission errors).

[0061] The predictor (635) may perform a prediction search for the coding engine (632). That is, for a new picture to be coded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc. that may serve as suitable prediction references for the new picture. The predictor (635) may operate using one sample block per pixel block to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (634).

[0062] The controller (650) can manage the encoding operations of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0063] The output of all the above mentioned functional units may be subjected to entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0064] The transmitter (640) can buffer the encoded video sequence created by the entropy coder (645) and prepare it for transmission over a communication channel (660), which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (640) can merge the encoded video data from the video coder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown).

[0065] The controller (650) can manage the operation of the video encoder (603). During encoding, the controller (650) can assign a particular encoding picture type to each encoded picture, which may affect the encoding technique that may be applied to the respective picture. For example, pictures are often assigned as one of the following picture types:

[0066] An intra picture (I-picture) may be one that can be coded and decoded without using other pictures in a sequence as a source of prediction. Some video codecs allow various types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of such variations of I-pictures, as well as their respective uses and characteristics.

[0067] A predictive picture (P picture) may be one that can be encoded and decoded using intra- or inter-prediction, using at most one motion vector and reference index to predict the sample values ​​of each block.

[0068] Bidirectionally predicted pictures (B-pictures) may be those that can be encoded and decoded using intra- or inter-prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0069] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the respective picture of the block. For example, blocks of an I-picture may be non-predictively coded, or they may be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of a P-picture may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.

[0070] The video encoder (603) may perform encoding operations according to a given video encoding technique or standard, such as ITU-T Rec. H.265. In its operations, the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancies in the input video sequence. Thus, the encoded video data may conform to a syntax specified by the video encoding technique or standard being used.

[0071] In one embodiment, the transmitter (640) can transmit additional data along with the coded video. The source coder (630) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0072] Video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation in a given picture, while inter-picture prediction exploits correlation (temporal or other) between pictures. In one example, a particular picture being coded / decoded, called the current picture, is divided into blocks. When a block in the current picture is similar to a reference block in a reference picture that was previously coded and is still buffered in the video, the block in the current picture can be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0073] In some embodiments, bi-prediction techniques can be used in inter-picture prediction. According to bi-prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture, both of which are before the current picture in the video in decoding order (but may be past and future, respectively, in display order). A block in the current picture can be coded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block can be predicted by a combination of the first reference block and the second reference block.

[0074] Furthermore, to improve coding efficiency, merge mode techniques can be used in inter-picture prediction.

[0075] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed within units of blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU includes three coding tree blocks (CTBs), which are one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree partitioned into one or more coding units (CUs). For example, a 64×64 pixel CTU can be partitioned into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one example, each CU is analyzed to determine the prediction type of the CU, such as an inter prediction type or an intra prediction type. The CU is partitioned into one or more prediction units (PUs) depending on the temporal and / or spatial predictability. In general, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation of coding (encoding / decoding) is performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values) of 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0076] 7 shows a diagram of a video encoder (703) according to another embodiment of the present disclosure. The video encoder (703) is configured to receive a processed block (e.g., a predictive block) of sample values ​​in a current video picture in a sequence of video pictures and to encode the processed block into a coded picture that is part of a coded video sequence. In one example, the video encoder (703) is used in place of the video encoder (403) of the example of FIG. 4.

[0077] In an HEVC example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a predictive block of 8×8 samples. The video encoder (703) determines whether the processing block is best coded using intra mode, inter mode, or bi-predictive mode, for example, using rate-distortion optimization. When the processing block is coded in intra mode, the video encoder (703) can code the processing block into a coded picture using intra prediction techniques, and when the processing block is coded in inter mode or bi-predictive mode, the video encoder (703) can code the processing block into a coded picture using inter prediction techniques or bi-predictive techniques, respectively. In certain video coding techniques, the merge mode may be an inter-picture prediction submode in which a motion vector is derived from one or more motion vector predictors without the benefit of coded motion vector components outside the predictors. In certain other video coding techniques, there may be motion vector components applicable to the current block. In one example, the video encoder (703) includes other components, such as a mode decision module (not shown), to determine the mode of the processing block.

[0078] In the example of FIG. 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together as shown in FIG.

[0079] The inter-encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block to one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a subsequent picture), generate inter-prediction information (e.g., a description of redundancy information, motion vectors, merge mode information according to an inter-coding technique), and calculate an inter-prediction result (e.g., a prediction block) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture that is decoded based on the coded video information.

[0080] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), possibly compare the block with already encoded blocks in the same picture, generate quantized coefficients after transformation, and possibly also generate intra prediction information (e.g., intra prediction direction information according to one or more intra encoding techniques). In one example, the intra encoder (722) also calculates intra prediction results (e.g., a prediction block) based on the intra prediction information and reference blocks in the same picture.

[0081] The generic controller (721) is configured to determine generic control data and control other components of the video encoder (703) based on the generic control data. In one example, the generic controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is an intra mode, the generic controller (721) controls the switch (726) to select an intra mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information and include the intra prediction information in the bitstream, and when the mode is an inter mode, the generic controller (721) controls the switch (726) to select an inter prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter prediction information and include the inter prediction information in the bitstream.

[0082] The residual calculator (723) is configured to calculate a difference (residual data) between the received block and a prediction result selected from the intra encoder (722) or the inter encoder (730). The residual encoder (724) is configured to operate based on the residual data to encode the residual data and generate transform coefficients. In one example, the residual encoder (724) is configured to transform the residual data from a spatial domain to a frequency domain and generate transform coefficients. The transform coefficients then undergo a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be used by the intra encoder (722) and the inter encoder (730) as appropriate. For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and the inter prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and the intra prediction information. The decoded blocks are suitably processed to generate a decoded picture, which may be buffered in a memory circuit (not shown) and may be used as a reference picture in some examples.

[0083] The entropy encoder (725) is configured to format the bitstream to include the encoded block. The entropy encoder (725) is configured to include various information according to an appropriate standard, such as the HEVC standard. In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information in the bitstream. It should be noted that, according to the disclosed subject matter, no residual information is present when encoding a block in a merged sub-mode of either the inter-mode or the bi-prediction mode.

[0084] 8 shows a diagram of a video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (810) is used in place of the video decoder (410) of the example of FIG. 4.

[0085] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter-decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-decoder (872), coupled together as shown in FIG.

[0086] The entropy decoder (871) may be configured to reconstruct from the coded picture certain symbols representing syntax elements that constitute the coded picture. Such symbols may include, for example, prediction information (e.g., intra prediction information or inter prediction information) that may identify the mode in which the block is coded (e.g., intra mode, inter mode, bi-predictive mode, merged sub-mode or the latter two of another sub-mode), certain samples or metadata used for prediction by the intra decoder (872) or inter decoder (880), respectively, residual information in the form of quantized transform coefficients, etc. In one example, when the prediction mode is an inter mode or bi-predictive mode, the inter prediction information is provided to the inter decoder (880), and when the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information may undergo inverse quantization and is provided to the residual decoder (873).

[0087] The inter decoder (880) is configured to receive the inter prediction information and to generate inter prediction results based on the inter prediction information.

[0088] The intra decoder (872) is configured to receive the intra prediction information and to generate a prediction result based on the intra prediction information.

[0089] The residual decoder (873) is configured to perform inverse quantization to extract inverse quantized transform coefficients, and process the inverse quantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (to include quantizer parameters (QP)), which may be provided by the entropy decoder (871) (this may be only a small amount of control information, so the data path is not depicted).

[0090] The reconstruction module (874) combines, in the spatial domain, the residual output by the residual decoder (873) and the prediction result (possibly output by an inter-prediction module or an intra-prediction module) to form a reconstructed block that may be part of a reconstructed picture, which may be part of the restored video. It should be noted that other suitable operations, such as a deblocking operation, may be performed to improve visual quality.

[0091] It should be noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technology. In one embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (603) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0092] Aspects of the present disclosure provide a technique of bi-prediction without signaling CU level weight. Bi-prediction refers to a technique in inter-prediction that uses two reference blocks that are respectively present in two reference pictures to perform prediction of a block in a current picture. In some examples, one of the two reference pictures is before the current picture in a video sequence, and the other of the two reference pictures is after the current picture in a video sequence, and bi-prediction is called bidirectional prediction. According to some aspects of the present disclosure, the bi-prediction technique uses a prediction model with CU level weight. The CU level weight can be calculated from previously decoded pixels, for example, by fixed-point arithmetic, and the CU level weight is not signaled in the bitstream.

[0093] In some related examples, bi-prediction with CU-level weights (BCW) may use one or more fixed weights that are signaled from the encoder side to the decoder side at the CU level (e.g., for each CU, respectively).

[0094] In one example (e.g., HEVC), a bi-predictive signal (sample) is generated by averaging two prediction signals (samples) obtained from two different reference pictures and / or using two different motion vectors. For example, to predict a sample in a current picture using bi-prediction, a first prediction signal is obtained by inter-prediction based on a first reference picture and a first motion vector, and a second prediction signal is obtained by inter-prediction based on a second reference picture and a second motion vector. Then, a bi-predictive signal in the current picture is obtained by averaging the first prediction signal and the second prediction signal. The weights in the averaging are fixed and are 0.5 for the first prediction signal and the second prediction signal, respectively.

[0095] In another example (e.g., VVC), the bi-prediction mode is extended beyond simple averaging to allow weighted averaging of two prediction signals. For example, P0[i,j] represents a first prediction signal of a pixel at a position (i,j) in the current CU, and P1[i,j] represents a second prediction signal of a pixel at a position (i,j) in the current CU. The first prediction signal is obtained by inter prediction based on a first reference picture and a first motion vector, and the second prediction signal is obtained by inter prediction based on a second reference picture and a second motion vector. P0[i,j] of a pixel at a position (i,j) is a first prediction signal of a pixel at a position (i,j) in the current CU. bi-pred The bi-predictive signal represented by [i,j] can be calculated according to equation (1). P bi-pred [i,j]=((8-w)×P0[i,j]+w×P1[i,j]+4)≫3 Equation (1) Here, w represents the weight of the second prediction signal. In some examples, the weight is signaled from the encoder side to the decoder side using a weight index. For example, in weighted average bi-prediction, five weights w∈{-2, 3, 4, 5, 10} are allowed. The BCW weight index is coded into the bitstream at the CU level, and the BCW weight index can indicate one of the five weights for using bi-prediction.

[0096] In some related examples, a technique called local illumination compensation (LIC) is used. In one example, LIC is used in inter prediction techniques for uni-prediction of inter-coded CUs. LIC does not apply to bi-prediction. LIC can model the local illumination variation between a current block and its reference block as a function of the variation between the current block template and the reference block template.

[0097] 9A-9C show diagrams illustrating uni-prediction of an inter-coded CU in some examples. In FIG. 9A, a current block (910) is located in a current picture for encoding. To inter-predict the current block (910), a reference block (920) in a reference picture is determined as a temporal prediction of the current block (910). The position of the reference block (920) in the reference picture may have a spatial shift from the current block (910) in the current picture, as indicated by the motion vector (930) in FIG. 9A.

[0098] 9B shows a current template and a reference template associated with a current block (910) and a reference block (920), respectively. In some examples, the current template of the current block (910) includes a top current template (911) and a left current template (912), as shown in FIG. 9B. The top current template (911) includes adjacent pixels above the current block (910) in the current picture. The left current template (912) includes adjacent pixels to the left of the current block (910) in the current picture.

[0099] Similarly, the reference templates of the reference block (920) include a top reference template (921) and a left reference template (922), as shown in Figure 9B. The top reference template (921) includes adjacent pixels above the reference block (920) in the reference picture. The left reference template (922) includes adjacent pixels to the left of the reference block (920) in the reference picture.

[0100] In some examples, the top template of a block includes the pixels of one row immediately above the block, and the left template of a block includes the pixels of one column immediately to the left of the block. As shown in FIG. 9B, the top current template (911) includes the pixels of one row immediately above the current block (910) in the current picture, and the left current template (912) includes the pixels of one column immediately to the left of the current block (910) in the current picture. The top reference template (921) includes the pixels of one row immediately above the reference block (920) in the reference picture. The left reference template (922) includes the pixels of one column immediately to the left of the reference block (920) in the reference picture.

[0101] In FIG. 9C, the reference block templates (including the top reference template (921) and the left reference template (922)) can be regarded as a temporal prediction of the current block templates (including the top current template (911) and the left current template (912)) indicated by the motion vector (930) between the current block (910) and the reference block (920).

[0102] In some examples, according to local illumination compensation (LIC), the samples in the current block can be predicted by applying the LIC model to the samples in the reference block. For example, the current block has a width w and a height h, and the reference block also has a width w and a height h. For ease of explanation, assume that the upper left corner of the current block has coordinates (0, 0), and assume that the upper left corner of the reference block needs to have coordinates (0, 0). Further, y[i,j] represents the sample value at the position (i,j) in the current block when 0≦i<w and 0≦j<h, and

Number

[0103] In some examples, the scale parameter α and the offset parameter β can be derived based on the current block template (e.g., (911) and (912)) and the reference block template (e.g., (921) and (922)). In one example, an LIC flag is signaled to indicate the use of LIC, and no signaling overhead is required for signaling the scale parameter α and the offset parameter β.

[0104] Note that LIC is an inter-prediction technique for single prediction of inter-coded CUs and is not applicable to bi-prediction.

[0105] According to one aspect of the present disclosure, BCW (e.g., VVC) in related examples limits the weight precision. In some examples, in weighted average bi-prediction, only a maximum of five weight values are allowed. Additionally, by signaling the weight index in the bitstream, the signaling overhead in the bitrate can be increased.

[0106] According to another aspect of the present disclosure, LIC can model the local illumination variation between the current block and the reference block of the current block as a function of the illumination variation between the current block template of the current block and the reference block template of the reference block. LIC does not add overhead in bit rate to signal the scale and offset parameters. However, LIC does not apply to bi-prediction.

[0107] Some aspects of this disclosure provide techniques for use in bi-prediction (eg, VVC) that use CU-level weights with higher weight precision without signaling the weights in the bitstream.

[0108] In some embodiments, in a bi-predictive mode, P0[i,j] represents a first prediction signal of a pixel at position (i,j) in the current CU, and P1[i,j] represents a second prediction signal of a pixel at position (i,j) in the current CU. bi-pred A bi-predictive signal, represented by [i,j], may be calculated as a weighted average of the first predicted signal and the second predicted signal, such as according to equation (3). P bi-pred [i,j]=((2 n -w) × P0[i,j] + w × P1[i,j] + 2 n-1 )≫n =P0[i,j]+(w·(P1[i,j]-P0[i,j])+2 n-1 )≫n Formula (3) where w represents a weight and n represents a predetermined shift parameter. In some examples, the weight w is an integer and ranges from 0 <w<2 n In one embodiment, when the first predicted signal P0 and the second predicted signal P1 are represented using 10 bits, n is set to 5, and therefore the calculation in equation (3) can be performed using 16-bit signed arithmetic.

[0109] According to some embodiments, the weight w may be determined based on a current template of the current CU and a reference template of the reference CU, and thus, in some examples, CU-level weight signaling is not required.

[0110] FIG. 10 shows a diagram of using bi-prediction to encode a video sequence (1000) according to some embodiments of this disclosure.

[0111] In the example of FIG. 10, a video sequence (1000) includes multiple picture frames, such as frames (1001)-(1003). Frame (1001) is the current frame for encoding, and frames (1002) and (1003) are coded (encoded or decoded) before frame (1001). In some examples, one of frames (1002) and (1003) is before frame (1001) in display order of the video sequence (1000), and the other of frames (1002) and (1003) is after frame (1001) in display order. Note that there may or may not be additional frames between frames (1002) and (1001), or between frames (1001) and (1003). In some other examples, frames (1002) and (1003) are both before frame (1001) in display order, or frames (1002) and (1003) are both after frame (1001) in display order.

[0112] In the example of FIG. 10, a current block (1010) is located in a current picture, such as frame (1001), for encoding. To inter predict the current block (1010) in a bi-predictive mode, a first reference block (1020) in a first reference picture, such as frame (1002), and a second reference block (1030) in a second reference picture, such as frame (1003), are determined. The position of the first reference block (1020) in frame (1002) may have a first spatial shift from the position of the current block (1010) in frame (1001), as indicated by a motion vector (1025) represented by mv0 in FIG. 10. The position of the second reference block (1030) in frame (1003) may have a second spatial shift from the position of the current block (1010) in frame (1001), as indicated by a motion vector (1035) represented by mv1 in FIG. 10. The first reference block (1020) may be referred to as a first prediction of the current block (1010) and may be denoted as P0. The second reference block (1030) may be referred to as a second prediction of the current block (1010) and may be denoted as P1. The current block (1010) may be predicted by a weighted average of the first prediction P0 and the second prediction P1.

[0113] In some embodiments, a current template of the current block (1020), a reference template of the first reference block (1020), and a reference template of the second reference block (1030) are used to determine the weight w in a bi-predictive mode.

[0114] In some examples, the current template of the current block (1010) is the current template T c The current template T c contains decoded pixels adjacent to the current block (1010). In some examples, the current template T cincludes the row of pixels immediately above the current block (1010) (e.g., designated (1011)) and the column of pixels immediately to the left of the current block (1010) (e.g., designated (1012)).

[0115] The reference template of the first reference block (1020) is referred to as the first reference template T0. The first reference template T0 includes decoded pixels adjacent to the first reference block (1020). In some examples, the first reference template T0 includes a row of pixels immediately above the first reference block (1020) (e.g., indicated by (1021)) and includes a column of pixels immediately to the left of the first reference block (1020) (e.g., indicated by (1022)).

[0116] The reference template of the second reference block (1030) is referred to as the second reference template T1. The second reference template T1 includes decoded pixels adjacent to the second reference block (1030). In some examples, the second reference template T1 includes a row of pixels immediately above the second reference block (1030) (e.g., indicated by (1031)) and includes a column of pixels immediately to the left of the second reference block (1030) (e.g., indicated by (1032)).

[0117] In some cases, the current template T c can have any suitable shape, and the current template T c It should be noted that the first reference template T0 and the second reference template T1 corresponding to can be determined based on the motion vectors mv0 and mv1.

[0118] According to one aspect of the present disclosure, when coding (encoding or decoding) a current block (1010), a current template T c The first reference picture (1002) and the second reference picture (1003) are reconstructed, and thus the first reference template T0 and the second reference template T1 are reconstructed. Then, the current template T c, the first reference template T0, and the second reference template T1, the weights w are determined based on the first reference template T0 and the second reference template T1. The weights w may be calculated at the encoder and the decoder, and thus in some examples, the weights w are not signaled in the bitstream.

[0119] It should be noted that the weights w may be calculated using any suitable technique.

[0120] In some embodiments, a cost function may be formed with the cost depending on the weight w. The weight w may then be determined as the argument that minimizes the cost of the cost function.

[0121] In some examples, the first reference template T0 and the second reference template T1 are calculated based on the predicted current template T w For example, the current template T c The predicted current template T at position (i,j) in w can be calculated according to equation (4). T w [i,j]=T0[i,j]+(w×(T1[i,j]-T0[i,j])+2 n-1 )≫n Formula (4)

[0122] In one embodiment, to calculate the cost function, T w [i,j] and T c The difference with [i,j] can be used.

[0123] In some examples, the cost function may be formed according to equation (5).

number

[0124] In some examples, the weights w are approximately determined by least squares fitting using fixed-point arithmetic and clipped to the range [a, b], where a and b are predefined integers.

[0125] After the weights w are determined, the current block (1010) can be predicted in a bi-predictive mode as a weighted average of the first prediction P0 and the second prediction P1, such as using equation (3).

[0126] 11 shows a flowchart outlining a process (1100) performed in a bi-predictive mode according to one embodiment of the present disclosure. The process starts at (S1101) and proceeds to (S1110).

[0127] In (S1110), a current block (e.g., the current block (1010)) is determined for encoding in a bi-predictive mode, and then a first motion vector mv0, a first reference picture, a second motion vector mv1, and a second reference picture are determined. Based on the first motion vector mv0 and the first reference picture, a first prediction P0 (e.g., the first reference block (1020)) is determined, and based on the second motion vector mv1 and the second reference picture, a second prediction P1 (e.g., the second reference block (1030)) is determined. Furthermore, the current template T c (e.g., (1011) and (1012)) are determined.

[0128] In (S1120), a first reference template T0 and a second reference template T1 are determined. In some examples, the first reference template T0 is a reference template T cand the second reference template T1 is determined based on the first motion vector mv0 and the current template T c and the second motion vector mv1.

[0129] At (S1130), a weight(s) for minimizing the cost of the cost function is determined. For example, the cost function can be formed based on Equation (4) and Equation (5) as a function of the weight w. Then, a weight w for minimizing the cost of the cost function is determined.

[0130] Then, at (S1140), the bi-prediction may be calculated as a weighted average of the first prediction P0 and the second prediction P1 based on the determined weight(s).

[0131] According to one aspect of this disclosure, multiple weighting parameters can be used in bi-predictive mode.

[0132] In some examples, three weight parameters are used in bi-prediction mode. For example, in bi-prediction mode, P of a pixel at position (i,j) in a current CU (e.g., current block (1010)) is bi-pred The bi-predictive signal represented by [i,j] can be calculated as a weighted average of the first predicted signal P0[i,j] and the second predicted signal P1[i,j] according to a prediction model such as the prediction model according to equation (6). P bi-pred [i,j]=(w0×P0[i,j]+w1×P1[i,j]+w2)≫n Equation (6) where w0, w1, and w2 are weight parameters (also called weights) of the prediction model, and n is a predetermined shift parameter. The weight parameters w0, w1, and w2 can be derived from templates (e.g., the current template, the first reference template, and the second reference template) by fixed-point arithmetic.

[0133] The values ​​of the weight parameters w0, w1, and w2 are the same as those of the current template T cand the first and second reference templates T0 and T1. In one example, the predicted current template T w can be formed as a function of the weighting parameters. For example, for the current template T c The predicted current template T at position (i,j) in w can be expressed according to equation (7). T w [i,j]=(w0×T0[i,j]+w1×T1[i,j]+w2)≫n Equation (7)

[0134] In some examples, the cost function may be w and the current template T c The cost function can be formed according to equation (5) using the weighting parameters w0, w1, and w2 such that the cost function is a function of the weighting parameters. In one example, the weighting parameters w0, w1, and w2 are such that 0≦T w [i,j]<2 b (Current template T c The cost of a cost function with the constraints that (for all positions in ), where a is the bit depth of the pixels, can be determined as the argument that minimizes the cost of the cost function with the constraints that (for all positions in ), where a is the bit depth of the pixels.

[0135] In one example, the weight parameters w0, w1, and w2 are approximately determined by a least squares method with fixed-point arithmetic using Cramer's rule. In some examples, the values ​​of the weight parameters can be clipped. In one example, w i is in the range [a i ,b i ] is clipped to a i and b i for a given integer i∈{0,1,2}.

[0136] In some examples, two weight parameters are used in bi-prediction mode. For example, in bi-prediction mode, P of a pixel at position (i,j) in a current CU (e.g., current block (1010)) is bi-predThe bi-predictive signal represented by [i,j] can be calculated as a weighted average of the first predicted signal P0[i,j] and the second predicted signal P1[i,j] according to a prediction model such as the prediction model according to equation (8). P bi-pred [i,j]=(w0×P0[i,j]+w1×P1[i,j]+C)≫n Equation (8) where w0, w1 are weight parameters (also called weights) of a prediction model that can be derived from templates (e.g., a current template, a first reference template, and a second reference template) by fixed-point arithmetic. Parameter C is a predetermined value for rounding, and n is a predetermined shift parameter.

[0137] The values ​​of the weight parameters w0 and w1 are the same as those of the current template T c and the first and second reference templates T0 and T1. In one example, the predicted current template T w can be formed as a function of the weighting parameters. For example, for the current template T c The predicted current template T at position (i,j) in the region w can be expressed according to equation (9). T w [i,j]=(w0×T0[i,j]+w1×T1[i,j]+C)≫n Equation (9)

[0138] In some examples, the cost function may be w and the current template T c The cost function can be formed according to equation (5) using the weighting parameters w0, w1, and the cost function is then a function of the weighting parameters. In one example, the weighting parameters w0, w1 are such that 0≦T w [i,j]<2 b (Current template T c The cost of a cost function with the constraints that (for all positions in ), where a is the bit depth of the pixels, can be determined as the argument that minimizes the cost of the cost function with the constraints that (for all positions in ), where a is the bit depth of the pixels.

[0139] In one example, the weight parameters w0, w1 are approximately determined by a least squares method using fixed-point arithmetic using Cramer's rule. In some examples, the values ​​of the weight parameters can be clipped. In one example, w i is in the range [a i ,b i ] is clipped to a i and b i for a given integer i∈{0,1}.

[0140] According to one aspect of the present disclosure, on the encoder side, the weighting parameter(s) to use in the bi-predictive mode may be determined together with the determination of the motion vectors mv0 and mv1.

[0141] FIG. 12 shows a flowchart outlining a process (1200) according to some embodiments of the present disclosure. In some examples, the process (1200) is used on the encoder side to jointly determine weight parameter(s) and a first motion vector mv0 and a second motion vector mv1 for use in a bi-predictive mode. The process (1200) performs motion vector refinement using an initial first motion vector mv0' and an initial second motion vector mv1' based on a predetermined search pattern. The process (1200) can use a predetermined bi-predictive model and a predetermined cost function based on fixed-point arithmetic to determine the best motion vectors mv0 and mv1 and the best weight(s). The best motion vectors mv0 and mv1 can be appropriately notified to the decoder side, and the best weight(s) are not signaled to the decoder side. The decoder side can determine the weights based on the notified motion vectors (e.g., the best motion vectors mv0 and mv1). The process starts at (S1201) and proceeds to (S1210).

[0142] In (S1210), the initial first motion vector mv0′, the initial second motion vector mv1′, and the current template T cis determined, a first refinement for mv0', denoted by dmv0, is initially set to 0, and a second refinement for mv1', denoted by dmv1, is initially set to 0. The initial first motion vector mv0' and the initial second motion vector mv1' may be determined by any suitable technique.

[0143] In (S1220), the refined first motion vector

number

number

[0144] In (S1230), the first reference template T0 is c and the refined first motion vector

number

number

[0145] In (S1240), a predetermined bi-prediction model (e.g., Equation (3), Equation (6), Equation (8)) having weighting parameter(s) is used to predict a template T based on a first reference template T0 and a second reference template T1 as shown in Equation (4), Equation (7), and Equation (9). w Therefore, the predicted template T wis expressed as a function of the weighting parameter(s). A given cost function is then calculated based on the predicted template T w and the current template T c can be used to represent a cost based on dmv0. Thus, the cost is a function of the weight parameter(s). Furthermore, the weight parameter(s) for minimizing the cost of the cost function are calculated. The minimum cost associated with the present refinements dmv0 and dmv1 is referred to as the new computational cost. The calculated weight parameter(s) are associated with the new computational cost. In some examples, the calculation can be performed by a least squares method with fixed-point arithmetic using Cramer's rule.

[0146] In (S1250), the new computational cost is compared to the previous best cost to update the best cost. In one example, when the new computational cost is smaller than the previous best cost, the new computational cost is stored as the best cost, the calculated weight parameter(s) associated with the new computational cost are retained as the best weight(s) associated with the best cost, and the current refined motion vector associated with the new computational cost is stored as the best weight parameter(s).

number

number

[0147] At (S1260), if the predetermined search pattern has been completely searched, the process proceeds to (S1280), otherwise the process proceeds to (S1270).

[0148] At (S1270), the first refinement dmv0 and the second refinement dmv1 are updated according to a predetermined search pattern, and the process returns to (S1220).

[0149] At (S1280), a first prediction P0 and a second prediction P1 are determined (eg, calculated) based on the best refined motion vector associated with the best cost.

[0150] At (S1290), a bi-prediction may be calculated based on the first prediction P0 associated with the best cost, the second prediction P1, and the best weight(s).

[0151] FIG. 13 shows a flow chart outlining a process (1300) according to one embodiment of the present disclosure. The process (1300) can be used in a video encoder. In various embodiments, the process (1300) is performed by a processing circuit, such as the processing circuit of the terminal devices (310), (320), (330), and (340), a processing circuit performing the function of the video encoder (403), a processing circuit performing the function of the video encoder (603), a processing circuit performing the function of the video encoder (703), etc. In some embodiments, the process (1300) is implemented with software instructions, and thus the processing circuit performs the process (1300) when the processing circuit executes the software instructions. The process starts at (S1301) and proceeds to (S1310).

[0152] At (S1310), a bi-prediction mode without weight signaling is determined for encoding the coding block in the current picture. In one example, the encoder can perform an evaluation and determine to use a bi-prediction mode with weight signaling to encode the coding block based on the evaluation.

[0153] At (S1320), motion vectors, such as a first motion vector associated with a first reference picture and a second motion vector associated with a second reference picture, as well as weights (one or more) for bi-prediction of the coding block are determined.

[0154] In some embodiments, the first motion vector and the second motion vector are determined, and then the weight(s) are determined. In some examples, after the motion vector is determined, a reference template in the reference picture is determined based on the current template and the motion vector of the coding block. For example, the first reference template in the first reference picture is determined based on the current template and the first motion vector of the coding block, and the second reference template in the second reference picture is determined based on the current template and the second motion vector of the coding block.

[0155] According to one aspect of the disclosure, a current template for a current block includes one or more reconstructed samples adjacent to a coding block. In some examples, the current template includes one or more rows of samples above the coding block and / or one or more columns of samples to the left of the coding block. In one example, the current template includes a row of samples immediately above the coding block and / or a column of samples immediately to the left of the coding block.

[0156] In some embodiments, the weight(s) to use in the bi-predictive mode are determined based on the first reference template, the second reference template, and the current template.

[0157] In some embodiments, the weight(s) is determined to minimize the cost of a predetermined cost function. The predetermined cost function is determined based on the first reference template, the second reference template, and the current template. For example, the predetermined cost function is based on each sample difference of the current template to the predicted current template, as in Equation (5).

[0158] The predicted current template is predicted based on the first reference template and the second reference template using a predefined bi-prediction model having one or more weighting parameters.

[0159] In some examples, the predetermined bi-predictive model includes three weighting parameters, and the predicted current template can be expressed, for example, according to Equation (7). In one example, the values ​​of the three weighting parameters in the predetermined bi-predictive model are determined by a least squares method using fixed-point arithmetic, such as using Cramer's rule.

[0160] In some examples, the predetermined bi-predictive model includes two weight parameters, and the predicted current template can be expressed, for example, according to Equation (9). In one example, the values ​​of the two weight parameters in the predetermined bi-predictive model are determined by a least squares method using fixed-point arithmetic, such as using Cramer's rule.

[0161] In some examples, the predetermined bi-predictive model includes one weight parameter, and the predicted current template can be expressed, for example, according to Equation (4). In one example, the value of the weight parameter in the predetermined bi-predictive model is determined by a least squares method using fixed-point arithmetic or the like.

[0162] In some examples, the value(s) of the weighting parameter are clipped within the respective ranges of the weighting parameter.

[0163] In some embodiments, the first motion vector, the second motion vector, and the weight(s) are determined jointly. An example for jointly determining the first motion vector, the second motion vector, and the weight(s) is described with reference to FIG.

[0164] At (S1330), the coding block is reconstructed using bi-prediction with weight(s). In some examples, the predetermined bi-prediction model includes three weight parameters, and the coding block can be reconstructed, for example, according to Equation (6). In some examples, the predetermined bi-prediction model includes two weight parameters, and the coding block can be reconstructed, for example, according to Equation (8). In some examples, the predetermined bi-prediction model includes one weight parameter, and the coding block can be reconstructed, for example, according to Equation (3).

[0165] In (S1340), the information of the coding block is coded in the bitstream without weight signaling. The information indicates a bi-prediction mode without weight signaling in the bitstream. Then, the process proceeds to (S1399) and ends.

[0166] The process (1300) may be adapted as appropriate. Step(s) of the process (1300) may be modified and / or omitted. Additional step(s) may be added. Any suitable order of performance may be used.

[0167] FIG. 14 shows a flow chart outlining a process (1400) according to one embodiment of the present disclosure. The process (1400) can be used in a video decoder. In various embodiments, the process (1400) is performed by a processing circuit, such as the processing circuit of the terminal devices (310), (320), (330), and (340), the processing circuit performing the function of the video decoder (410), the processing circuit performing the function of the video decoder (510), etc. In some embodiments, the process (1400) is implemented in software instructions, and thus the processing circuit performs the process (1400) when the processing circuit executes the software instructions. The process starts at (S1401) and proceeds to (S1410).

[0168] At (S1410), information of a coded block in a current picture is decoded from a bitstream, the information indicating a bi-prediction mode without weight signaling.

[0169] At (S1420), a motion vector for bi-prediction of the coding block is determined. For example, a motion vector such as a first motion vector associated with a first reference picture and a second motion vector associated with a second reference picture is determined. The motion vector can be determined by any suitable technique. In some examples, the motion vector is determined based on an index to a candidate list, and the index can be decoded from the bitstream.

[0170] At (S1430), a reference template in a reference picture is determined based on a current template and a motion vector of the coding block. For example, a first reference template in a first reference picture is determined based on a current template and a first motion vector of the coding block, and a second reference template in a second reference picture is determined based on a current template and a second motion vector of the coding block.

[0171] According to one aspect of the disclosure, a current template for a current block includes one or more reconstructed samples adjacent to a coding block. In some examples, the current template includes one or more rows of samples above the coding block and / or one or more columns of samples to the left of the coding block. In one example, the current template includes a row of samples immediately above the coding block and / or a column of samples immediately to the left of the coding block.

[0172] At (S1440), weight(s) to use in the bi-predictive mode are determined based on the first reference template, the second reference template, and the current template.

[0173] In some embodiments, the weights are determined to minimize the cost of a predetermined cost function. The predetermined cost function is determined based on the first reference template, the second reference template, and the current template. For example, the predetermined cost function is based on the sample difference of the current template relative to the predicted current template, such as by using Equation (5).

[0174] The predicted current template is predicted based on the first reference template and the second reference template using a predefined bi-prediction model having one or more weighting parameters.

[0175] In some examples, the predetermined bi-predictive model includes three weighting parameters, and the predicted current template can be expressed, for example, according to Equation (7). In one example, the values ​​of the three weighting parameters in the predetermined bi-predictive model are determined by a least squares method using fixed-point arithmetic, such as using Cramer's rule.

[0176] In some examples, the predetermined bi-predictive model includes two weight parameters, and the predicted current template can be expressed, for example, according to Equation (9). In one example, the values ​​of the two weight parameters in the predetermined bi-predictive model are determined by a least squares method using fixed-point arithmetic, such as using Cramer's rule.

[0177] In some examples, the predetermined bi-predictive model includes one weight parameter, and the predicted current template can be expressed, for example, according to Equation (4). In one example, the value of the weight parameter in the predetermined bi-predictive model is determined by a least squares method using fixed-point arithmetic or the like.

[0178] In some examples, the value(s) of the weighting parameter are clipped within the respective ranges of the weighting parameter.

[0179] At (S1450), the coding block is reconstructed using bi-prediction with weight(s). In some examples, the predetermined bi-prediction model includes three weight parameters, and the coding block can be reconstructed, for example, according to Equation (6). In some examples, the predetermined bi-prediction model includes two weight parameters, and the coding block can be reconstructed, for example, according to Equation (8). In some examples, the predetermined bi-prediction model includes one weight parameter, and the coding block can be reconstructed, for example, according to Equation (3).

[0180] The process then proceeds to (S1499) and ends.

[0181] It should be noted that in some examples, when any of the coding block's neighbors are unavailable, pre-defined weights (eg, default weights, etc.) may be used.

[0182] Process (1400) may be adapted as appropriate. Step(s) of process (1400) may be modified and / or omitted. Additional step(s) may be added. Any suitable order of performance may be used.

[0183] The techniques described above may be implemented as computer software using computer-readable instructions physically stored on one or more computer-readable media. For example, FIG. 15 illustrates a computer system (1500) suitable for implementing certain embodiments of the disclosed subject matter.

[0184] The computer software may be coded using any suitable machine code or computer language that may undergo assembly, compilation, linking, or similar mechanisms to produce code containing instructions that may be executed, such as by one or more computer central processing units (CPUs) and graphics processing units (GPUs), either directly, or through interpretation and execution of microcode.

[0185] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0186] 15 with respect to computer system (1500) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having a dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system (1500).

[0187] The computer system (1500) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more users, such as, for example, tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, clapping, etc.), visual input (gestures, etc.), olfactory input (not shown), etc. The human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (voice, music, ambient sounds, etc.), images (scanned images, photographic images obtained from still image cameras, etc.), video (two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0188] The input human interface devices may include one or more (only one of each is shown) of a keyboard (1501), a mouse (1502), a trackpad (1503), a touch screen (1510), a data glove (not shown), a joystick (1505), a microphone (1506), a scanner (1507), and a camera (1508).

[0189] The computer system (1500) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the user's senses, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (1510), data gloves (not shown), or joystick (1505), although there may also be haptic feedback devices that do not function as input devices), audio output devices (such as speakers (1509), headphones (not shown)), visual output devices (such as screens (1510), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may output two-dimensional visual output or three or more dimensional output via means such as stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0190] The computer system (1500) may also include human accessible storage devices and their associated media, such as optical media, including CD / DVD ROM / RW (1520) with media (1521) such as CDs / DVDs, thumb drives (1522), removable hard drives or solid state drives (1523), legacy magnetic media such as tapes and floppy disks (not shown), and dedicated ROM / ASIC / PLD based devices such as security dongles (not shown).

[0191] Also, those skilled in the art should understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0192] The computer system (1500) may also include an interface (1554) to one or more communication networks (1555). The network may be, for example, wireless, wired, optical. The network may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial including CANBus, etc. Certain networks typically require an external network interface adapter attached to a particular general-purpose data port (e.g., a USB port of the computer system (1500)) or peripheral bus (1549), while other networks are typically integrated into the core of the computer system (1500) by attaching to a system bus described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1500) can communicate with other entities. Such communications may be one-way receive only (e.g., broadcast TV), one-way transmit only (e.g., CANbus to a specific CANbus device), or bidirectional with other computer systems using, for example, local or wide area digital networks. Specific protocols and protocol stacks may be used with each of these networks and network interfaces, as described above.

[0193] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to the core (1540) of the computer system (1500).

[0194] The cores (1540) may include one or more central processing units (CPUs) (1541), graphics processing units (GPUs) (1542), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (1543), hardware accelerators for specific tasks (1544), graphics adapters (1550), and the like. These devices may be connected via a system bus (1548), along with read-only memory (ROM) (1545), random access memory (1546), internal mass storage devices (1547) such as internal hard drives, SSDs, etc. that are not accessible to the user. In some computer systems, the system bus (1548) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripherals may be connected directly to the core's system bus (1548) or via a peripheral bus (1549). In one example, a screen (1510) may be connected to the graphics adapter (1550). Peripheral bus architectures include PCI, USB, and the like.

[0195] The CPU (1541), GPU (1542), FPGA (1543), and accelerator (1544) can execute certain instructions that, in combination, can constitute the aforementioned computer code. This computer code can be stored in a ROM (1545) or a RAM (1546). Persistent data can be stored, for example, in an internal mass storage device (1547), while transient data can also be stored in the RAM (1546). Fast storage and retrieval from any of the memory devices can be enabled by the use of a cache memory, which can be closely associated with one or more of the CPU (1541), GPU (1542), mass storage (1547), ROM (1545), RAM (1546), etc.

[0196] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the type well known and available to those skilled in the computer software arts.

[0197] As an example and not by way of limitation, the architecture, particularly the computer system (1500) having the core (1540), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be the user-accessible mass storage devices introduced above, as well as media associated with specific storage of the core (1540) of a non-transitory nature, such as the core internal mass storage (1547) or ROM (1545). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (1540). The computer-readable media can include one or more memory devices or chips, depending on the particular needs. The software can cause the core (1540), and specifically the processor therein (including a CPU, GPU, FPGA, etc.) to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (1546) and modifying such data structures according to the software-defined processes. Additionally or alternatively, the computer system may achieve functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (1544)) that may operate in place of or together with software to perform certain processes or certain portions of certain processes described herein. Reference to software may include logic, and vice versa, as appropriate. Reference to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that stores software for execution, circuitry that embodies logic for execution, or both, as appropriate. The present disclosure encompasses any suitable combination of hardware and software. Appendix A: Abbreviations JEM Collaborative Exploration Model VVC Versatile Video Coding BMS Benchmark Set MV Motion Vector HEVC High Efficiency Video Coding SEI Supplemental Extension Information VUI Video Usability Information GOP Group of Pictures TU Conversion Unit PU Prediction Unit CTU Coding Tree Unit CTB Coding Tree Block PB Predicted Block HRD Hypothetical Reference Decoder SNR Signal to Noise Ratio CPU Central Processing Unit GPU Graphics Processing Unit CRT Braun tube LCD Liquid Crystal Display OLED Organic Light Emitting Diode CD Compact Disc DVD Digital Video Disc ROM Read Only Memory RAM Random Access Memory ASIC Application Specific Integrated Circuit PLD Programmable Logic Device LAN Local Area Network GSM Global System for Mobile Communications LTE Long Term Evolution CANBus Controller Area Network Bus USB Universal Serial Bus PCI Peripheral Component Interconnect FPGA Field Programmable Gate Area SSD Solid State Drive IC Integrated Circuit CU Coding Unit

[0198] While this disclosure has described several exemplary embodiments, there are modifications, substitutions, and various substitute equivalents which are within the scope of this disclosure. It will thus be appreciated that those skilled in the art can devise numerous systems and methods which, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope. [Explanation of symbols]

[0199] 101, 202-206 Samples 102, 103 Arrows 104 Block 201, 910, 1010 Current Block 310, 320, 330, 340 Terminal Devices 350 Network 300, 400 Communication Systems 401, 601 video sources 402 Stream 404 Encoded Video Data 405 Streaming Server 406, 408 Client Subsystem 407, 409 Copying of encoded video data 410, 510, 810 Video Decoder 411 Output Stream 412 Display 413 Ingestion Subsystem 420, 430, 530, 620 Electronic Devices 501 Channel 512 Rendering Device 515 Buffer Memory 520 Entropy Decoder / Parser 521 Symbols 531 Receiver 551 Scaler / Descaler Unit 552 Intra-picture prediction unit 553 Motion Compensation Prediction Unit 555 Aggregator 556 Loop Filter Unit 557, 634 Reference Picture Memory 558 Current Picture Buffer 603, 703 Video Encoder 630 Source Coder 633 (local) decoder 634 Reference Picture Cache 635 Predictor 640 Transmitter 643 coded video sequence 645 Entropy Coder 650 Controller 660 Communication Channels 721 General-purpose controller 722 Intra Encoder 723 Residual Calculator 724 Residual Encoder 725 Entropy Encoder 726 Switch 728, 873 Residual Decoder 730 InterEncoder 871 Entropy Decoder 872 Intra Decoder 874 Reconstruction Module 880 Interdecoder Current template on 911 912 Current template on the left 920 Reference Block 921 above reference template 922 Left Reference Template 930, 1035 motion vectors 1000 Video Sequences 1001~1003 Frames 1011, 1012, 1031, 1032 pixels 1020 First Reference Block 1030 Second Reference Block 1200, 1300, 1400 processes

Claims

1. 1. A method of video processing in a decoder, comprising the steps of: Decoding information of a coding block in a current picture from a bitstream, the information indicating a VVC-based bi-prediction mode, and a CU level weight associated with the bi-prediction mode is not signaled in the bitstream; determining a first motion vector associated with a first reference picture and a second motion vector associated with a second reference picture for bi-prediction of the coding block; determining a first reference template in the first reference picture based on a current template of the coding block and the first motion vector; determining a second reference template of the second reference picture based on the current template of the coding block and the second motion vector; calculating the CU level weight by a least squares method using fixed-point arithmetic based on the first reference template, the second reference template, and the current template, where the CU level weight is n bits; clipping the CU level weights within a predetermined range to between a and b, where a and b are integers; reconstructing the coding block using bi-prediction with the CU level weights; Including, The bi-predictive signal calculated based on the CU level weight includes a first predicted signal and a second predicted signal in the current picture; The first prediction signal and the second prediction signal have a bit length of m, and n and m are set so that the sum of n and m satisfies a predetermined number of bits; The current template is two or more rows of samples above the coding block; or Two or more columns of left samples of the coding block Contains at least one of method.

2. The method of claim 1 , wherein the current template comprises one or more reconstructed samples adjacent to the coding block.

3. The current template is the row of samples immediately above said coding block, or The sequence of samples immediately to the left of the coding block The method of claim 2, comprising at least one of:

4. The step of calculating the CU level weights further comprises: determining the CU level weights that minimize a predetermined cost function based on the first reference template, the second reference template, and the current template; The method of claim 1, further comprising:

5. 5. The method of claim 4, wherein the predetermined cost function is based on respective sample differences of the current template relative to a predicted current template predicted based on the first reference template and the second reference template using a predetermined bi-prediction model having one or more weighting parameters.

6. The step of calculating the CU level weights further comprises: determining values ​​of three weighting parameters in a given bi-predictive model by the least squares method; The method of claim 1, further comprising:

7. The step of calculating the CU level weights further comprises: determining values ​​of two weight parameters in a given bi-predictive model by the least squares method; The method of claim 1, further comprising:

8. The step of calculating the CU level weights further comprises: determining the value of one weight parameter in the given bi-predictive model by the least squares method; The method of claim 1, further comprising:

9. Apparatus configured to carry out the method according to any one of claims 1 to 8.

10. A program for causing at least one processor to carry out the method according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • Systems and methods for generalized multi-hypothesis prediction for video coding

    JP2019519148A

  • System and method for adaptively determining template size for illumination compensation

    JP2019531029A

  • Encoding method, decoding method, encoding device, and decoding device

    WO2014002217A1