Implicit mask mixing mode

Abstract

A computing system receives a video bitstream including a current block and syntax elements indicating that the current block is predicted using a mixed mode. The current block is encoded using information from the first reference block and the second reference block. When a portion of a current block corresponds to a first region, the system reconstructs the portion by averaging a reference value of a first reference block and a reference value of a second reference block, the first region: (i) being within a corresponding reference boundary of the first reference block and a corresponding reference boundary of the second reference block, or (ii) is not within a corresponding reference boundary of the first reference block and a corresponding reference boundary of the second reference block. When a portion of the current block corresponds to a second region, the system derives weighted reference values and reconstructs the portion by combining the weighted reference values, the second region being located only within one of the corresponding reference boundaries of the first reference block and the corresponding reference boundaries of the second reference block.

Description

Cross-references

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 461,879, filed on April 25, 2023, entitled “Implicit Masked Blending Mode,” and is a continuation-in-part of, and claims the benefit of priority to, U.S. Provisional Patent Application No. 18 / 241,757, filed on September 1, 2023, entitled “Implicit Masked Blending Mode.” Technical Field

[0002] The disclosed embodiments relate generally to video coding and decoding, including but not limited to systems and methods for block portions of linear and non-linear blends in wedge-based prediction modes. Background Art

[0003] Digital video is supported by various electronic devices, such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smart phones, video teleconferencing devices, video streaming devices, etc. Electronic devices send and receive or otherwise transmit digital video data over a communication network, and / or store digital video data on a storage device. Due to the limited bandwidth capacity of a communication network and the limited storage resources of a storage device, video encoding may be used to compress the video data according to one or more video encoding standards before transmitting or storing the video data. Video encoding may be performed by hardware and / or software on an electronic device / client device or server providing a cloud service.

[0004] Video coding typically utilizes a prediction method (e.g., inter-frame prediction, intra-frame prediction, etc.) that exploits the redundancy inherent in video data. Video coding aims to compress video data into a form that uses a lower bit rate while avoiding or minimizing the degradation of video quality. A variety of video codec standards have been developed. For example, High-Efficiency Video Coding (HEVC / H.265) is a video compression standard designed as part of the MPEG-H project. ITU-T and ISO / IEC released the HEVC / H.265 standard in 2013 (version 1), 2014 (version 2), 2015 (version 3), and 2016 (version 4). Versatile Video Coding (VVC / H.266) is a video compression standard that is a successor to HEVC. ITU-T and ISO / IEC released the VVC / H.266 standard in 2020 (version 1) and 2022 (version 2). AOMedia Video 1 (AV1) is an open video coding format designed to replace HEVC. On January 8, 2019, the verified version 1.0.0 with the errata 1 of the specification was released. Summary of the invention

[0005] The present disclosure describes the use of implicit mask blending modes (e.g., boundary-aware composite prediction) for blending block portions. For example, in a composite prediction block, a reference block from a reference picture may at least partially extend beyond a boundary. In some systems, the portion of the reference block that extends beyond the boundary is padded and therefore does not reflect the true pixel values, which may result in less accurate predictions. The disclosed methods and systems provide more accurate reconstruction based on true pixel data (e.g., giving greater weight to true pixel data), thereby improving encoding accuracy (e.g., by reducing artifacts in decoded video data).

[0006] According to some embodiments, a video decoding method is provided. The method includes: (a) receiving a video code stream, the video code stream including a current picture and a current block in the current picture; (b) determining, based on syntax elements in the video code stream, that the current block is encoded using information from a first reference block and a second reference block; (c) determining that at least one of the first reference block and the second reference block is at least partially located outside a corresponding reference boundary; (d) based on determining that a portion of the current block corresponds to a first region, reconstructing the portion by averaging a reference value of the first reference block and a reference value of the second reference block, the first region: (i) located at a corresponding reference boundary of the first reference block and The present invention relates to a method for reconstructing a portion of the current block according to the present invention, wherein the portion of the current block corresponds to a second area and the second area is located within only one of the corresponding reference boundaries of the first reference block and the second reference block: (i) deriving a weighted reference value by applying corresponding weights to the reference value of the first reference block and the reference value of the second reference block; and (ii) reconstructing the portion by combining the weighted reference value of the first reference block and the weighted reference value of the second reference block.

[0007] According to some embodiments, a video encoding method is provided. The method includes: (a) receiving a current picture and a current block within the current picture; (b) determining to use information from a first reference block and a second reference block to encode the current block; (c) determining that at least one of the first reference block and the second reference block is at least partially located outside a corresponding reference boundary; (d) encoding the portion by averaging a reference value of the first reference block and a reference value of the second reference block based on determining that the portion of the current block corresponds to a first region, the first region (i) being located within the corresponding reference boundary of the first reference block and the corresponding reference boundary of the second reference block, or (ii) not being located within the corresponding reference boundary of the first reference block and the corresponding reference boundary of the second reference block; and (e) based on determining that the portion of the current block corresponds to a second region, the second region being located within only one of the corresponding reference boundary of the first reference block and the corresponding reference boundary of the second reference block: (i) deriving a weighted reference value by applying corresponding weights to the reference value of the first reference block and the reference value of the second reference block; and (ii) encoding the portion by combining the weighted reference value of the first reference block and the weighted reference value of the second reference block.

[0008] According to some embodiments, a computing system is provided, such as a streaming system, a server system, a personal computer system, or other electronic device. The computing system includes a control circuit and a memory, and the memory stores one or more instruction sets. The one or more instruction sets include instructions for executing any method described in the present disclosure. In some embodiments, the computing system includes an encoder component and a decoder component (e.g., a transcoder component).

[0009] According to some embodiments, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores one or more instruction sets executed by a computing system. The one or more instruction sets include instructions for executing any method described in the present disclosure.

[0010] Therefore, methods, devices and systems for encoding and decoding video are disclosed. Such methods, devices and systems can supplement or replace traditional methods, devices and systems for video encoding / decoding.

[0011] The features and advantages described in the specification are not necessarily all-inclusive, and in particular, some additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, description, and claims provided in the present disclosure. In addition, it should be noted that the language used in the specification is primarily selected for readability and instructional purposes and is not necessarily selected to describe or limit the subject matter described in the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to understand the present disclosure in more detail, a more specific description may be made by reference to the features of various embodiments, some of which are shown in the accompanying drawings. However, the attached drawings only illustrate the relevant features of the present disclosure and are therefore not necessarily considered to be limiting, as those skilled in the art will understand when reading the present disclosure that the description may admit other effective features.

[0013] Figure 1 is a block diagram illustrating an exemplary communication system in accordance with some embodiments.

[0014] Figure 2A is a block diagram illustrating exemplary elements of an encoder assembly according to some embodiments.

[0015] Figure 2B is a block diagram illustrating exemplary elements of a decoder component according to some embodiments.

[0016] Figure 3 is a block diagram illustrating an exemplary server system in accordance with some embodiments.

[0017] FIG. 4A to FIG. 4D An exemplary coding tree structure according to some embodiments is shown.

[0018] Figure 5A An example of a partition-based prediction mode according to some embodiments is shown.

[0019] Figure 5B-Figure 5C An exemplary partitioning mode mix is ​​shown in accordance with some embodiments.

[0020] Figure 5D An exemplary wedge-based partitioning is shown in accordance with some embodiments.

[0021] Figure 5E-5F An exemplary scenario is shown in which a reference block is at least partially located outside the boundary of a corresponding reference picture according to some embodiments.

[0022] Fig. 6A is a flow chart illustrating an exemplary method of video encoding according to some embodiments.

[0023] Figure 6B is a flow chart illustrating an exemplary method of video decoding according to some embodiments.

[0024] As is common practice, the various features shown in the drawings are not necessarily drawn to scale, and like reference numerals may be used to denote like features throughout the specification and drawings. DETAILED DESCRIPTION

[0025] Among other things, the present disclosure describes the use of various partitioning techniques to partition video blocks to obtain optimal motion prediction and higher quality encoding. The present disclosure also describes the use of implicit mask blending (e.g., boundary-aware composite prediction) to reconstruct a current block that is encoded using information from two reference blocks through a composite mode. As used herein, the "implicit mask blending" refers to a mask blending that is not explicitly signaled in a video bitstream. As described in more detail below, the composite mode can be an effective inter-frame prediction coding tool. In addition, the combination of optical flow-based prediction and / or temporal interpolated prediction (TIP) mode can amplify the capabilities of the composite mode. However, in some cases, one or both of the two reference blocks extend beyond the corresponding image boundaries. In this case, samples outside the picture boundary can be filled, which is suboptimal for the composite mode because the prediction is no longer based on real pixel data. The use of implicit mask blending can improve the encoding accuracy of the composite mode (e.g., reducing artifacts in the decoded video data) by providing a more accurate reconstruction based on the real pixel data (e.g., giving greater weight to the real pixel data), compared to the current design, in which portions outside the boundaries of the reference block are filled and do not reflect the real pixel values. Exemplary Systems and Apparatus

[0026] Figure 1 1 is a block diagram illustrating a communication system 100 according to some embodiments. The communication system 100 includes a source device 102 and a plurality of electronic devices 120 (e.g., electronic device 120-1 to electronic device 120-m) communicatively coupled to each other via one or more networks. In some embodiments, the communication system 100 is a streaming system, such as for video-enabled applications, such as video conferencing applications, digital television applications, and media storage and / or distribution applications.

[0027] Source device 102 includes a video source 104 (e.g., a camera component or a media storage) and an encoder component 106. In some embodiments, video source 104 is a digital camera (e.g., configured to create an uncompressed video sample stream). Encoder component 106 generates one or more encoded video streams from the video stream. The video stream from video source 104 can be high data volume compared to the encoded video stream 108 generated by encoder component 106. Because the encoded video stream 108 is a lower data volume (less data) than the video stream from the video source, the encoded video stream 108 requires less bandwidth for transmission and requires less storage space for storage compared to the video stream from video source 104. In some embodiments, source device 102 does not include encoder component 106 (e.g., configured to transmit uncompressed video data to network 110).

[0028] The one or more networks 110 represent any number of networks that transmit information between the source device 102, the server system 112, and / or the electronic device 120, including, for example, wired (wired) and / or wireless communication networks. The one or more networks 110 can exchange data in circuit switching and / or packet switching channels. Representative networks include telecommunication networks, local area networks, wide area networks, and / or the Internet.

[0029] One or more networks 110 include a server system 112 (e.g., a distributed / cloud computing system). In some embodiments, the server system 112 is or includes a streaming server (e.g., configured to store and / or distribute video content, such as an encoded video stream from a source device 102). The server system 112 includes a codec component 114 (e.g., configured to encode and / or decode video data). In some embodiments, the codec component 114 includes an encoder component and / or a decoder component. In various embodiments, the codec component 114 is instantiated as hardware, software, or a combination thereof. In some embodiments, the codec component 114 is configured to decode the encoded video stream 108 and re-encode the video data using different encoding standards and / or methods to generate encoded video data 116. In some embodiments, the server system 112 is configured to generate multiple video formats and / or encodings based on the encoded video stream 108.

[0030] In some embodiments, the server system 112 functions as a Media-Aware Network Element (MANE). For example, the server system 112 may be configured to prune the encoded video stream 108 to tailor potentially different streams to one or more electronic devices 120. In some embodiments, the MANE is provided separately from the server system 112.

[0031] The electronic device 120-1 includes a decoder component 122 and a display 124. In some embodiments, the decoder component 122 is configured to decode the encoded video data 116 to generate an output video stream that can be presented on a display or other type of presentation device. In some embodiments, one or more electronic devices 120 do not include a display component (e.g., communicatively coupled to an external display device and / or include a media storage device). In some embodiments, the electronic device 120 is a streaming client. In some embodiments, the electronic device 120 is configured to access the server system 112 to obtain the encoded video data 116.

[0032] The source device and / or multiple electronic devices 120 are sometimes referred to as “end devices” or “user devices.” In some embodiments, the source device 102 and / or one or more electronic devices 120 are examples of server systems, personal computers, portable devices (e.g., smartphones, tablets, or laptops), wearable devices, video conferencing devices, and / or other types of electronic devices.

[0033] In an exemplary operation of the communication system 100, the source device 102 transmits the encoded video code stream 108 to the server system 112. For example, the source device 102 can encode a stream of pictures collected by the source device. The server system 112 receives the encoded video code stream 108 and can decode and / or encode the encoded video code stream 108 using a codec component 114. For example, the server system 112 can apply encoding to the video data, which is more optimal for network transmission and / or storage. The server system 112 can transmit the encoded video data 116 (e.g., one or more encoded video code streams) to one or more electronic devices 120. Each electronic device 120 can decode the encoded video data 116 to recover and optionally display the video picture.

[0034] In some embodiments, the transmission discussed above is a unidirectional data transmission. The unidirectional data transmission is sometimes used in applications such as media service applications. In some embodiments, the transmission discussed above is a bidirectional data transmission. Bidirectional data transmission is sometimes used in applications such as video conferencing applications. In some embodiments, the encoded video stream 108 and / or the encoded video data 116 are encoded and / or decoded according to any video encoding / compression standard described in the present disclosure, such as HEVC, VVC and / or AV1.

[0035] Figure 2A 1 is a block diagram illustrating example elements of an encoder component 106 according to some embodiments. The encoder component 106 receives a source video sequence from a video source 104. In some embodiments, the encoder component includes a receiver (e.g., a transceiver) component configured to receive the source video sequence. In some embodiments, the encoder component 106 receives a video sequence from a remote video source (e.g., a video source of a component of a device different from the encoder component 106). The video source 104 may provide the source video sequence in the form of a digital video sample stream, which may be any suitable bit depth (e.g., 8, 10, or 12 bits), any color space (e.g., BT.601 Y CrCB or RGB), and any suitable sampling structure (e.g., Y CrCb 4:2:0 or Y CrCb 4:4:4). In some embodiments, the video source 104 is a storage device that stores previously captured / prepared video. In some embodiments, the video source 104 is a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate pictures that are given motion when viewed sequentially. The image itself can be constructed as a spatial pixel array, where each pixel may include one or more samples depending on the sampling structure, color space, etc. used. The relationship between pixels and samples can be easily understood by those skilled in the art. The following description focuses on samples.

[0036] The encoder component 106 is configured to encode and / or compress the pictures of the source video sequence into an encoded video sequence 216 in real time or under any other time constraints required by the application. Implementing the appropriate encoding speed is a function of the controller 204. In some embodiments, the controller 204 controls other functional units as described below and is functionally coupled to these units. The parameters set by the controller 204 may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization techniques, etc.), picture size, picture group (Group Of Pictures, GOP) layout, maximum motion vector search range, etc. Other functions of the controller 204 can be easily identified by those of ordinary skill in the art because these other functions involve the video encoder component 106 optimized for a certain system design.

[0037] In some embodiments, the encoder component 106 is configured to operate in an encoding loop. In a simplified example, the encoding loop may include a source encoder 202 (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder 210. The decoder 210 reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (when any compression between the symbols and the encoded video code stream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory 208. Since the decoding of the symbol stream produces bit-accurate results that are independent of the decoder location (local or remote), the contents of the reference picture memory 208 are also bit-accurate between the local encoder and the remote encoder. In this way, the reference picture samples interpreted by the prediction part of the encoder are exactly the same as the sample values ​​that the decoder will interpret when using the prediction during decoding. This principle of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, such as due to channel errors) is known to those of ordinary skill in the art.

[0038] The operation of decoder 210 may be combined with, for example, Figure 2B The remote decoder described in detail, such as decoder assembly 122, is the same. However, briefly referring to Figure 2B , when symbols are available and the entropy encoder 214 and the parser 254 are capable of losslessly encoding / decoding the symbols into an encoded video sequence, the entropy decoding portion of the decoder component 122, including the buffer memory 252 and the parser 254, may not be fully implemented in the local decoder 210.

[0039] Except parsing / entropy decoding, the decoder technology described in the present disclosure can be present in the corresponding encoder in the form of substantially the same function. For this reason, the present disclosure focuses on the decoder operation. The description of the encoder technology can be simplified because the encoder technology is mutually inverse to the decoder technology described comprehensively. Only a more detailed description is needed in certain areas, and it is provided below.

[0040] As part of the operation, the source encoder 202 may perform motion compensated predictive coding that predictively encodes an input frame with reference to one or more previously encoded frames from a video sequence designated as "reference frames." In this manner, the encoding engine 212 encodes the differences between pixel blocks of an input frame and pixel blocks of a reference frame that may be selected as a prediction reference for the input frame. The controller 204 may manage the encoding operations of the source encoder 202, including, for example, the setting of parameters and subgroup parameters for encoding video data.

[0041] The decoder 210 may decode the encoded video data of the frame that may be designated as the reference frame based on the symbol created by the source encoder 202. The operation of the encoding engine 212 may advantageously be a lossy process. Figure 2A When decoded at a remote video decoder (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. Decoder 210 replicates the decoding process that may be performed by the video decoder on the reference frame and may cause the reconstructed reference frame to be stored in reference picture cache 208. In this way, encoder component 106 may locally store a copy of the reconstructed reference frame that has common content (absent transmission errors) with the reconstructed reference frame that will be obtained by the remote video decoder.

[0042] The predictor 206 may perform a prediction search for the encoding engine 212. That is, for a new frame to be encoded, the predictor 206 may search the reference picture memory 208 for sample data (as a candidate reference pixel block) or certain metadata, such as a reference picture motion vector, block shape, etc., that may serve as a suitable prediction reference for the new picture. The predictor 206 may operate pixel-by-pixel based on sample blocks to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor 206, it may be determined that the input picture may have a prediction reference taken from a plurality of reference pictures stored in the reference picture memory 208.

[0043] The outputs of all the above functional units may be entropy encoded in the entropy encoder 214. The entropy encoder 214 performs lossless compression on the symbols generated by the various functional units according to techniques known to those skilled in the art (e.g., Huffman coding, variable length coding, arithmetic coding), thereby converting the symbols into a coded video sequence.

[0044] In some embodiments, the output of the entropy encoder 214 is coupled to a transmitter. The transmitter can be configured to buffer the encoded video sequence created by the entropy encoder 214, thereby preparing for transmission through a communication channel 218, which can be a hardware / software link to a storage device that will store the encoded video data. The transmitter can be configured to merge the encoded video data from the source encoder 202 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown). In some embodiments, the transmitter can transmit additional data when transmitting the encoded video. The source encoder 202 can use such data as part of the encoded video sequence. Additional data may include redundant data in other forms such as time / space / SNR enhancement layers, redundant pictures and slices, auxiliary enhancement information (Supplementary Enhancement Information, SEI) messages, visual usability information (VUI) parameter set fragments, etc.

[0045] The controller 204 may manage the operation of the encoder component 106. During encoding, the controller 204 may assign a certain encoded picture type to each encoded picture, but this may affect the encoding techniques that may be applied to the corresponding picture. For example, a picture may typically be assigned as an intra picture (I picture), a predictive picture (P picture), or a bidirectional predictive picture (B picture). An intra picture may be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art understand the variants of I pictures and their corresponding applications and features, so this disclosure will not be repeated. Predictive pictures may be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block. Bidirectional predictive pictures may be encoded and decoded using intra prediction or inter prediction, which uses at most two motion vectors and reference indexes to predict the sample values ​​of each block. Similarly, multiple predictive pictures may use more than two reference pictures and associated metadata for reconstructing a single block.

[0046] The source picture may typically be spatially subdivided into blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples), and coded block by block. These blocks may be predictively coded with reference to other (already coded) blocks, which are determined according to the coding allocation applied to the block's corresponding picture. For example, a block of an I picture may be non-predictively coded, or the block may be predictively coded (spatial prediction or intra-frame prediction) with reference to an already coded block of the same picture. A block of a P picture may be non-predictively coded by spatial prediction with reference to one previously coded reference picture or by temporal prediction. A block of a B picture may be non-predictively coded by spatial prediction with reference to one or two previously coded reference pictures or by temporal prediction.

[0047] The captured video may be taken as a plurality of source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often simplified to intra-prediction) exploits spatial correlations in a given picture, while inter-picture prediction exploits (temporal or other) correlations between pictures. In an embodiment, a particular picture being encoded / decoded is divided into blocks, and the particular picture being encoded / decoded is referred to as the current picture. When a block in the current picture is similar to a reference block in a reference picture that was previously encoded in the video and is still buffered, the block in the current picture may be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference picture, and in the case where multiple reference pictures are used, the motion vector may have a third dimension that identifies the reference picture.

[0048] The encoder component 106 can perform encoding operations according to a predetermined video encoding technique or standard, such as any of the techniques or standards described in the present disclosure. In operation, the encoder component 106 can perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data can conform to the syntax specified by the video encoding technique or standard used.

[0049] Figure 2B is a block diagram illustrating exemplary elements of decoder component 122 according to some embodiments. Figure 2B The decoder component 122 in is coupled to the channel 218 and the display 124. In some embodiments, the decoder component 122 includes a transmitter coupled to the loop filter 256 and configured to transmit data to the display 124 (eg, via a wired or wireless connection).

[0050] In some embodiments, the decoder component 122 includes a receiver coupled to the channel 218 and configured to receive data from the channel 218 (e.g., via a wired or wireless connection). The receiver can be configured to receive one or more encoded video sequences for decoding by the decoder component 122. In some embodiments, the decoding of each encoded video sequence is independent of other encoded video sequences. Each encoded video sequence can be received from the channel 218, which can be a hardware / software link to a storage device storing the encoded video data. The receiver can receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that can be forwarded to their respective use entities (not shown). The receiver can separate the encoded video sequence from the other data. In some embodiments, the receiver can receive additional (redundant) data together with the encoded video. The additional data is included as part of the encoded video sequence. The additional data can be used by the video decoder 122 to decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0051] According to some embodiments, the decoder component 122 includes: a buffer memory 252, a parser 254 (sometimes also referred to as an entropy decoder), a scaler / inverse transform unit 258, an intra picture prediction unit 262, a motion compensated prediction unit 260, an aggregator 268, a loop filter unit 256, a reference picture memory 266, and a current picture memory 264. In some embodiments, the decoder component 122 is implemented as an integrated circuit, a series of integrated circuits, and / or other electronic circuits. In some embodiments, the decoder component 122 is at least partially implemented in software.

[0052] Buffer memory 252 is coupled between channel 218 and parser 254 (e.g., to prevent network jitter). In some embodiments, buffer memory 252 is separate from decoder component 122. In some embodiments, a separate buffer memory is provided between the output of channel 218 and decoder component 122. In some embodiments, in addition to buffer memory 252 internal to decoder component 122 (e.g., which is configured to handle playout timing), a separate buffer memory is provided external to decoder component 122 (e.g., to prevent network jitter). Buffer memory 252 may not be required or may be made smaller when data is received from a store / forward device with sufficient bandwidth and controllability or from an isochronous network. Buffer memory 252 may be necessary for use on a traffic packet network such as the Internet, and may be relatively large and may have an adaptive size and may be implemented at least in part in an operating system or similar element (not shown) external to decoder component 122.

[0053] The parser 254 is configured to reconstruct the symbol 270 from the coded video sequence. The symbol may include, for example, information for managing the operation of the decoder component 122, and / or information for controlling a display device such as the display 124. The control information for the display device may be in the form of, for example, auxiliary enhancement information (SEI message) or video usability information (VUI) parameter set fragments (not shown). The parser 254 parses (entropy decodes) the coded video sequence. The encoding of the coded video sequence may be performed according to a video coding technique or standard, and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser 254 may extract a subgroup parameter set for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), and the like. Parser 254 may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[0054] Depending on the type of the coded video picture or a portion of the coded video picture (e.g., inter-frame and intra-frame pictures, inter-frame blocks and intra-frame blocks) and other factors, the reconstruction of the symbol 270 may involve multiple different units. Which units are involved and how they are involved may be controlled by subgroup control information parsed from the coded video sequence by the parser 254. For the sake of brevity, such subgroup control information flow between the parser 254 and the multiple units is not described.

[0055] The decoder component 122 may be conceptually subdivided into several functional units. And in some embodiments, these units closely interact with each other and may be at least partially integrated with each other. However, for the sake of clarity, this conceptual subdivision into functional units is maintained.

[0056] The sealer / inverse transform unit 258 receives the quantized transform coefficients and control information (e.g., which transform method to use, block size, quantization factor, and / or quantization scaling matrix) as symbols 270 from the parser 254. The sealer / inverse transform unit 258 may output a block including sample values, which may be input into the aggregator 268.

[0057] In some cases, the output samples of the sealer / inverse transform unit 258 belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed pictures, but can use predictive information from previously reconstructed portions of the current picture. Such predictive information may be provided by the intra-picture prediction unit 262. The intra-picture prediction unit 262 uses the reconstructed information obtained from the current picture memory 264 to generate surrounding blocks of the same size and shape as the block being reconstructed. The aggregator 268 adds the prediction information generated by the intra-prediction unit 262 to the output sample information provided by the sealer / inverse transform unit 258 on a per-sample basis.

[0058] In other cases, the output samples of the sealer / inverse transform unit 258 may belong to an inter-frame coded block and a potential motion compensated block. In this case, the motion compensated prediction unit 260 may access the reference picture memory 266 to extract samples for prediction. After the extracted samples are motion compensated according to the symbol 270 associated with the block, these samples may be added to the output of the sealer / inverse transform unit 258 (in this case, referred to as residual samples or residual signals) by the aggregator 268 to generate output sample information. The extraction of prediction samples by the motion compensated prediction unit 260 from the address in the reference picture memory 266 may be controlled by a motion vector, and the motion vector is provided to the motion compensated prediction unit 260 in the form of the symbol 270, which includes, for example, X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​extracted from the reference picture memory 266 when using sub-sample accurate motion vectors, motion vector prediction mechanisms, and the like.

[0059] The output samples of aggregator 268 may be employed by various loop filtering techniques in loop filter unit 256. The video compression techniques may include in-loop filter techniques controlled by parameters included in the encoded video bitstream and available to loop filter unit 256 as symbols 270 from parser 254. However, the video compression techniques may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of an encoded picture or encoded video sequence, as well as to previously reconstructed and loop filtered sample values.

[0060] The output of the loop filter unit 256 may be a sample stream that may be output to a display device such as the display 124 , as well as stored in the reference picture memory 266 for subsequent inter-picture prediction.

[0061] Once fully reconstructed, certain coded pictures may be used as reference pictures for future prediction. Once a coded picture is fully reconstructed, and the coded picture is identified as a reference picture (e.g., by parser 254), the current reference picture may become part of reference picture memory 266, and a new current picture memory may be reallocated before starting reconstruction of a subsequent coded picture.

[0062] The decoder component 122 may perform decoding operations according to a predetermined video compression technique. The technique may be recorded in a standard, such as any standard described in the present disclosure. In the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the profile recorded in the video compression technique or standard, the encoded video sequence may conform to the syntax specified by the video compression technique or standard used. In addition, in order to conform to some video compression techniques or standards, the complexity of the encoded video sequence is within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum picture size, the maximum frame rate, the maximum reconstruction sampling rate (measured in, for example, mega samples per second), the maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further defined by the Hypothetical Reference Decoder (HRD) specification and the metadata of the HRD buffer management represented by the signal in the encoded video sequence.

[0063] Figure 3is a block diagram illustrating a server system 112 according to some embodiments. The server system 112 includes a control circuit 302, one or more network interfaces 304, a memory 314, a user interface 306, and one or more communication buses 312 for interconnecting these components. In some embodiments, the control circuit 302 includes one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a data processing unit (DPU)). In some embodiments, the control circuit includes one or more field programmable gate arrays (FPGA), hardware accelerators, and / or one or more integrated circuits (e.g., application specific integrated circuits).

[0064] The network interface 304 can be configured to interface with one or more communication networks (e.g., wireless networks, wired networks, and / or optical networks). The communication network can be a local network, a wide area network, a metropolitan area network, a vehicle and industrial network, a real-time network, a delay-tolerant network, and the like. Examples of communication networks include local area networks such as Ethernet, wireless LANs, cellular networks including Global System for Mobile communications (GSM), 3G, 4G, 5G, Long-Term Evolution (LTE), etc., television wired or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicle and industrial networks including CANBus, and the like. Such communications can be one-way reception only (e.g., broadcast television), one-way transmission only (e.g., CANbus to certain CANbus devices), or bidirectional, for example, using a local area network or a wide area network digital network to connect to other computer systems. Such communications can include communications to one or more cloud computing networks.

[0065] The user interface 306 includes one or more output devices 308 and / or one or more input devices 310. The input device 310 may include one or more of the following: a keyboard, a mouse, a touch pad, a touch screen, a data glove, a joystick, a microphone, a scanner, a camera. The output device 308 may include one or more of the following: an audio output device (e.g., a speaker), a visual output device (e.g., a display or monitor), etc.

[0066] The memory 314 may include high-speed random access memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM), double data rate (DDR) RAM and / or other random access solid-state memory devices) and / or non-volatile memory (e.g., one or more magnetic disk storage devices, optical disk storage devices, flash memory devices and / or other non-volatile solid-state storage devices). The memory 314 optionally includes one or more storage devices remote from the control circuit 302. The memory 314, or alternatively, the non-volatile solid-state memory device within the memory 314, includes a non-transitory computer-readable storage medium. In some embodiments, the memory 314 or the non-transitory computer-readable storage medium of the memory 314 stores the following programs, modules, instructions and data structures, or a subset or superset thereof: Operating system 316, including processes for handling various basic system services and for performing hardware-related tasks; A network communication module 318 for connecting the server system 112 to other computing devices via one or more network interfaces 304 (e.g., via wired and / or wireless connections); A codec module 320 for performing various functions related to encoding and / or decoding data (eg, video data). In some embodiments, the codec module 320 is an example of the codec component 114. The codec module 320 includes but is not limited to: o a decoding module 322 for performing various functions related to decoding encoded data, such as those previously described with respect to the decoder component 122; and o an encoding module 340 for performing various functions related to encoding data, such as those previously described with respect to the encoder component 106; and Picture memory 352 for storing pictures and picture data, for example for use with the codec module 320. In some embodiments, the picture memory 352 includes one or more of the following: reference picture memory 208, buffer memory 252, current picture memory 264, and reference picture memory 266.

[0067] In some embodiments, the decoding module 322 includes a parsing module 324 (e.g., configured to perform the various functions previously described with respect to the parser 254), a transform module 326 (e.g., configured to perform the various functions previously described with respect to the scaler / inverse transform unit 258), a prediction module 328 (e.g., configured to perform the various functions previously described with respect to the motion compensated prediction unit 260 and / or the intra-picture prediction unit 262), and a filter module 330 (e.g., configured to perform the various functions previously described with respect to the loop filter 256).

[0068] In some embodiments, the encoding module 340 includes an encoding module 342 (e.g., configured to perform various functions previously described with respect to the source encoder 202 and / or the encoding engine 212) and a prediction module 344 (e.g., configured to perform various functions previously described with respect to the predictor 206). In some embodiments, the decoding module 322 and / or the encoding module 340 include Figure 3 A subset of the modules shown. For example, both the decoding module 322 and the encoding module 340 use a shared prediction module.

[0069] Each of the above-identified modules stored in the memory 314 corresponds to an instruction set for performing the functions described in the present disclosure. The above-identified modules (e.g., instruction sets) do not need to be implemented as separate software programs, processes, or modules, so in various embodiments, various subsets of these modules can be combined or otherwise rearranged. For example, the codec module 320 optionally does not include a separate decoding module and encoding module, but uses the same set of modules to perform two sets of functions. In some embodiments, the memory 314 stores subsets and data structures of the above-identified modules. In some embodiments, the memory 314 stores additional modules and data structures not described above, such as audio processing modules.

[0070] In some embodiments, the server system 112 includes a web page or hypertext transfer protocol (HTTP) server, a file transfer protocol (FTP) server, and web pages and applications implemented using common gateway interface (CGI) scripts, PHP hypertext preprocessor (PHP), active server pages (ASP), hypertext markup language (HTML), extensible markup language (XML), Java, JavaScript, asynchronous JavaScript and XML (AJAX), XHP, Javelin, Wireless Universal Resource File (WURFL), etc.

[0071] although Figure 3 The server system 112 is shown according to some embodiments, but Figure 3The present disclosure is intended more as a functional description of various features that may be present in one or more server systems rather than as a block diagram of the embodiments described in the present disclosure. In practice, as will be appreciated by one of ordinary skill in the art, items shown separately may be combined and some items may be separated. For example, Figure 3 Some of the items shown individually in the figure may be implemented on separate servers, and a single item may be implemented by one or more servers. The actual number of servers used to implement server system 112 and the manner in which features are distributed among them will vary from implementation to implementation, and may depend in part on the amount of data traffic that the server system handles during peak usage and during average usage. Exemplary Encoding and Decoding Processes and Techniques

[0072] The encoding processes and techniques described below may be performed on the devices and systems described above (eg, source device 102, server system 112, and / or electronic device 120). FIG. 4A to FIG. 4D An exemplary coding tree structure according to some embodiments is shown. Figure 4A As shown in the first coding tree structure (400) in FIG, some coding methods (e.g., VP9) use a 4-way partition tree from the 64×64 level down to the 4×4 level, and have some additional restrictions for 8×8 blocks. Figure 4A The partitioning designated as R in can be called recursive because the same partitioning tree is repeated at a lower scale until the lowest 4×4 level is reached.

[0073] like Figure 4B As shown in the second coding tree structure (402) in FIG. 4 , some coding methods (e.g., AV1) extend the partition tree to a 10-way structure and increase the maximum size (e.g., superblock in VP9 / AV1 parlance) to start at 128×128. The second coding tree structure includes 4:1 / 1:4 rectangular partitions that are not present in the first coding tree structure. Figure 4B The partition type with 3 sub-partitions in the second row of is called T-type partition. In addition to the coding block size, a coding tree depth can be defined to indicate the partition depth starting from the root node.

[0074] For example, a Coding Tree Unit (CTU) can be divided into Coding Units (CUs) using a quadtree structure represented as a coding tree to accommodate various local characteristics, such as in HEVC. In some embodiments, the decision on whether to encode a picture area using inter (temporal) prediction or intra (spatial) prediction is made at the CU level. Depending on the partition type of the PU, each CU can be further split into one PU, two PUs, or four PUs. Within a PU, the same prediction process is applied, and the relevant information is transmitted to the decoder PU by PU. After obtaining the residual block by applying the prediction process based on the PU partition type, the CU can be divided into TUs according to another quadtree structure similar to the coding tree for the CU.

[0075] For example, in VVC, a quadtree with nested multi-type trees using a binary and ternarysplits segmentation structure can replace the concept of multiple partition unit types. For example, this type of quadtree removes the concept of distinguishing between CU, PU, ​​and TU, unless the size of the CU is too large for the maximum transform length and the distinction is required, and this type of quadtree supports greater flexibility in the shape of CU partitions. In the coding tree structure, the CU can have a square or rectangular shape. The CTU is first divided by a quadtree (also called a quadtree) structure. The quadtree leaf nodes can be further divided by a multi-type tree structure. Figure 4C As shown in the third coding tree structure (404) in FIG, the multi-type tree structure includes four types of partitioning. The multi-type tree leaf node is called a CU, and this partitioning is used for prediction and transform processing without any further division unless the CU is too large for the maximum transform length. This means that in most cases, in a quadtree with a nested multi-type tree coding block structure, CU, PU, ​​and TU have the same block size. Figure 4D An example of block partitioning for one CTU (406) is shown, showing an example quadtree with a nested multi-type tree coding block structure.

[0076] Motion estimation involves determining motion vectors, which describe the transformation from one image (picture) to another. Reference images (or blocks) can come from adjacent frames in a video sequence. Motion vectors can be related to the entire image (global motion estimation) or to a specific block. In addition, motion vectors can correspond to translation models or distortion models that approximate motion (e.g., rotation and translation in three dimensions, and scaling). By further partitioning the blocks, motion estimation can be improved in some cases (e.g., with more complex video objects).

[0077] The geometric partitioning mode (GPM) can focus on inter-frame prediction blocks (e.g., CUs). When GPM is applied to a block, the block is divided into two parts via a straight partition boundary. The position of the partition boundary can be mathematically defined by an angle parameter φ and an offset parameter ρ. These parameters can be quantized and combined into a GPM partition index lookup table. The GPM partition index of the current block can be encoded into the bitstream. For example, for a CU of size w×h=2k×2l (in terms of luma samples), where k, l∈{3…6}, the GPM in VVC supports 64 partitioning modes. For CUs with aspect ratios greater than 4:1 or less than 1:4, for example, because narrow CUs rarely contain geometric separation patterns, the GPM can be disabled.

[0078] After the partitioning, the two GPM parts (partitions) contain separate motion information that can be used to predict the corresponding parts in the current block. In some embodiments, for each part of the GPM, only unidirectional motion compensated prediction (MCP) is allowed so that the memory bandwidth required for MCP in the GPM is equal to the memory bandwidth for conventional bidirectional MCP. In order to simplify the motion information encoding and reduce the possible combinations of GPMs, the motion information can be encoded in a merge mode. The GPM merge candidate list can be derived from the merge candidate list to ensure that only unidirectional motion information is included.

[0079] Figure 5A The prediction process of the GPM according to some embodiments is shown. The current block 510 is divided into a right part and a left part via a partition 516. The right prediction part (e.g., CU) of the current block 510 of the current picture 502 (e.g., size w×h) is predicted by MV0 of a reference block 512 from a reference picture 504, while the left part is predicted by MV1 of a reference block 514 from a reference picture 506.

[0080] In some embodiments, when the current block is divided into two partitions via a partition (e.g., partition 516), a blend (e.g., corresponding to a "blending mode" or "mask blending") is applied to combine the two partitioned blocks. Blending the partitions involves applying a weighted sum of the prediction values ​​of the respective partitions. Figure 5B An exemplary mixing matrix for a partition (e.g., partition 516) according to some embodiments is shown. In this example, the final GPM prediction (PG) is generated by performing a mixing process using integer mixing matrices W0 and W1 (e.g., containing weights in the range of 0 to 8). This can be expressed as: Where W0+W1=8J Formula 1- Hybrid Prediction In Formula 1, J is a matrix of size w×h consisting of ones. The weights of the mixing matrix may depend on the displacement between the sample positions and the partition boundaries. The computational complexity of the mixing matrix derivation may be low, so that these matrices may be generated dynamically at the decoder side.

[0081] The generated GPM prediction (PG) can then be subtracted from the original signal to generate a residual. For example, the residual can be transformed, quantized and encoded into the bitstream using a conventional VVC transform, quantization and entropy coding engine. On the decoder side, the signal is reconstructed by adding the residual to the GPM prediction PG. For example, when the residual is negligible, GPM can also support a skip mode. For example, the encoder discards the residual and the decoder directly uses the GPM prediction PG as the reconstructed signal.

[0082] Wedge-based prediction is an example of a hybrid mode. Wedge-based prediction (for example, in AV1) is a composite prediction mode that is similar to GPM. Wedge-based prediction can be used for inter-inter combination as well as inter-intra combination. The boundaries of moving objects are usually difficult to approximate by block partitioning on a grid. One solution is to predefine a codebook of possible wedge partitions (for example 16) when the coding unit is further divided in such a way, and to signal the wedge index in the codestream. In the current wedge design in AV1, since a maximum of 16 symbols can be written in one syntax element using multi-symbol adaptive context coding used in AV1, 16 modes are supported. As Figure 5D As shown, a 16-ary shape codebook is designed for the square block 540 and the rectangular block 542, and the 16-ary shape codebook includes a partition direction of horizontal, vertical or inclined (e.g., a slope of ±2 or ±0.5). In order to reduce the spurious high-frequency components usually generated by directly juxtaposing two predictors, a 2-D wedge mask with a gentle cliff shape can be used to smooth the edges around the expected partition (e.g., m(i, j) is close to 0.5 around the edge and gradually converted to binary weights at either end).

[0083] The wedge mode in AV1 can be extended to allow the wedge mode to be used for 64×64 blocks, 32×64 blocks, 64×32 blocks, 16×64 blocks, and 64×16 blocks. Figure 5C As shown, the wedge pattern can be defined in the Hessian norm form, where the angle Indicates the direction of the partition boundary, and the distance p indicates the offset of the partition boundary from the center of the block. The angle can be quantized into a value (e.g., 20 values) using a tangent value. The distance can be quantized based on the block size. For example, three distances can be used for angles greater than or equal to 180 degrees and angles of 0 or 90 degrees. For other angles, four distances can be used. In this way, 8×4+12×3=68 modes can be supported. Since more than 16 modes are supported, three syntax elements can be used to signal the wedge index, such as angle direction, angle, and distance. The angle direction indicates whether the angle is less than 180 degrees. Based on the angle direction, the actual angle can be signaled. Based on the angle signaled, the distance can be signaled.

[0084] The wedge blend mask can be directly quantized based on the distance from the sample position to the segmentation boundary. Using the Hessian norm form of the segmentation boundary definition, the distance can be defined as: Formula 2 - Blending Mask Distance where ρ is the distance from the center, is the partition angle. The angle and distance can be quantized by using the tangent value and the block size. Therefore, as shown in the following formula 3, only the lookup table and the shift operation can be used to calculate the quantized d(m,n). Formula 3- Quantized distance

[0085] Formula 4 can be used to derive the mixing weights at the corresponding positions: ω m,n = clamp(0,64,(d(m,n)+32)) Formula 4 - Mixing Weight

[0086] The blending weights may be calculated dynamically (eg, due to low computational complexity), or may be pre-stored (eg, with the AV1 wedge mode design).

[0087] In some embodiments, the composite prediction block is predicted by two or more reference blocks, each reference block having a corresponding reference picture. Some current systems use a simple average or a weighted average to combine the values ​​from the two reference blocks to generate the composite prediction block. In some cases, at least one of the reference blocks is at least partially outside the boundary. Figure 5E and Fig. 5F An exemplary scenario is shown according to some embodiments, in which a composite prediction block is predicted using a reference block that is partially outside the boundary. Figure 5EIn , the composite prediction block is predicted using the value of the reference block 556 from the reference picture 552 and the value of the reference block 558 from the reference picture 554. The shaded area of ​​the reference block 556 is located within the corresponding reference boundary 566 (e.g., the reference picture boundary), while the non-shaded area of ​​the reference block 556 is located outside the corresponding reference boundary 566. The shaded area of ​​the reference block 558 is located within the corresponding reference boundary 568, while the non-shaded area of ​​the reference block 556 is located outside the corresponding reference boundary 568. Fig. 5F , the composite prediction block is predicted using the value of a reference block 576 from a reference picture 572 and the value of a reference block 578 from a reference picture 574. The shaded area of ​​the reference block 576 is within the corresponding reference boundary 586, while the non-shaded area of ​​the reference block 576 is outside the corresponding reference boundary 586. The shaded area of ​​the reference block 578 is within the corresponding reference boundary 588, while the non-shaded area of ​​the reference block 578 is outside the corresponding reference boundary 588. In some current systems, the portion outside the boundary of the reference block is filled and therefore does not reflect the actual pixel value, which may result in less accurate prediction.

[0088] In some embodiments, an implicit mask is generated based on determining that at least one predictor of the composite mode is partially or completely outside the corresponding image boundary. In some embodiments, the mask is generated using the example code snippet described below, and the mask composite mode is applied. x0=x+mv0 x ;y0=y+mv0 y ; x1=x+mv1 x ;y1=y+mv1 y is_P (x1,y1) _out=x1<0 or x1≥frame_width or y1<0 or y1≥frame_height else m(x,y)=32 Among them, (x, y) represents the sample position in the image coordinate system; and is_P (x1,y1) _out is a variable indicating whether the samples at (x0, y0) and (x1, y1) are outside the corresponding boundaries; and m(x, y) represents the weight factor of the generated mask at (x, y).

[0089] In some embodiments, as shown in Formula 5, a prediction value P(x, y) is generated for the mask composite mode. P(x,y)=clip((m(x,y)*P0(x,y)+(64-m(x,y))*P1(x,y)+32)>>6) Formula 5-Composite model prediction value

[0090] In some embodiments, the implicit mask generation process described above is applied to the composite mode, the optical flow-based composite mode and / or the temporal interpolation prediction (TIP) mode. In some embodiments, for the optical flow (including the optical flow for TIP) mode, the implicit mask generation process is only applied to the final prediction signal after the refinement process, and the implicit mask generation process is not involved during the refinement.

[0091] The methods and processes described below detail the implicit mask blending mode. According to some embodiments, the implicit mask blending mode is implemented when at least a portion of a reference block used to predict a current block is outside a corresponding boundary (e.g., a reference picture boundary). The implicit mask blending mode can be used with the GPM and / or wedge-based prediction process described above. Turning now to some exemplary embodiments.

[0092] Fig. 6A is a flow chart illustrating a video encoding method 600 according to some embodiments. The method 600 may be executed on a computing system (e.g., server system 112, source device 102, or electronic device 120) having control circuitry and a memory storing instructions executed by the control circuitry. In some embodiments, the method 600 is executed by executing instructions stored in a memory (e.g., memory 314) of the computing system. The system receives (602) a current picture and a current block within the current picture. The system determines (604) to encode the current block using information from a first reference block and a second reference block. The system determines (606) that at least one of the first reference block and the second reference block is at least partially located outside a corresponding reference boundary. Based on determining that a portion of the current block corresponds to a first region, the system encodes (608) the portion by averaging a reference value of the first reference block and a reference value of the second reference block, the first region: (i) being located within a corresponding reference boundary of the first reference block and a corresponding reference boundary of the second reference block (e.g., Figure 5E 560) or (ii) is not located within the corresponding reference boundary of the first reference block and the corresponding reference boundary of the second reference block (e.g., Figure 5E According to determining that the portion of the current block corresponds to a second region, the second region is located only within one of the corresponding reference boundaries of the first reference block and the corresponding reference boundaries of the second reference block (e.g., Figure 5Eregion 562 in the image), the system (i) derives (610) a weighted reference value by applying corresponding weights to the reference value of the first reference block and the reference value of the second reference block; and (ii) encodes the portion by combining the weighted reference value of the first reference block and the weighted reference value of the second reference block.

[0093] Figure 6B 6 is a flow chart illustrating a video decoding method 650 according to some embodiments. The method 650 may be performed on a computing system (e.g., server system 112, source device 102, or electronic device 120) having control circuitry and a memory storing instructions executed by the control circuitry. In some embodiments, the method 650 is performed by executing instructions stored in a memory (e.g., memory 314) of the computing system. The system receives (652) a video stream, the video stream comprising a current picture and a current block within the current picture. In some embodiments, the system receives a syntax element indicating that the current block is to be predicted in a hybrid mode. In some embodiments, based on the syntax element in the video stream, the system determines (654) that the current block is encoded using information from a first reference block and a second reference block. The system determines (656) that at least one of the first reference block and the second reference block is at least partially located outside a corresponding reference boundary. Based on determining that the portion of the current block corresponds to a first region, the system reconstructs the portion by averaging a reference value of the first reference block and a reference value of the second reference block (658), the first region (i) being within a corresponding reference boundary of the first reference block and a corresponding reference boundary of the second reference block (e.g., region 560), or (ii) not being within a corresponding reference boundary of the first reference block and a corresponding reference boundary of the second reference block (e.g., region 564). Based on determining that the portion of the current block corresponds to a second region, the second region being within only one of a corresponding reference boundary of the first reference block and a corresponding reference boundary of the second reference block (e.g., region 562), the system derives (660) a weighted reference value by applying corresponding weights to the reference value of the first reference block and the reference value of the second reference block; and (ii) reconstructs the portion by combining the weighted reference value of the first reference block and the weighted reference value of the second reference block.

[0094] Although Fig. 6A and 6BThe various logical stages are shown in a particular order, but stages that are not dependent on order may be reordered, and other stages may be combined or decomposed. Some reordering or other groupings not specifically mentioned will be apparent to one of ordinary skill in the art, and thus the ordering and groupings given in this disclosure are not exhaustive. Furthermore, it should be appreciated that the stages may be implemented in hardware, firmware, software, or any combination thereof.

[0095] In some embodiments, some / shaded areas are within the reference picture boundary, while other / transparent areas are outside the reference picture boundary. Simple averaging is used to average the common areas outside the dashed lines (i.e., both reference predictor p0 and reference predictor p1 are within the reference picture boundary, or both are outside the reference picture boundary), while implicit mask weighting is used to combine the areas between the dashed lines (i.e., only one reference predictor is within the reference picture, while the other reference block is outside the reference picture). For example, Figure 5E Reference block 556 and reference block 558 (or Fig. 5F , where reference block 576 and reference block 578 are located at a corner of reference picture 574), the shaded area of ​​the reference block is located within the corresponding boundary of the corresponding reference picture (e.g., reference picture 552 and reference picture 554), while the non-shaded area is located outside the corresponding boundary of the corresponding reference picture. In some embodiments, a simple average (e.g., an unweighted average) is used to average the common area (e.g., area 560) of reference block 556 and reference block 558 located within the corresponding reference boundary or the common area (e.g., area 564) of reference block 556 and reference block 558 located outside the corresponding reference boundary. In some embodiments, implicit mask weighting is used to combine an area (e.g., area 562) including (a) one reference block located within the corresponding reference boundary and (b) another reference block located outside the corresponding reference boundary, which means that the corresponding weights are not signaled in the video bitstream (e.g., the corresponding weights are derived at the decoder component). For example, see B1 described below.

[0096] In some embodiments, the weight factors are fixed and the sum of the weight factors is equal to 2 to the power of n. For example, the weight factor of the first predictor P0 (e.g., corresponding to reference block 556) can be represented as w1. The weight factor of the second predictor P1 (e.g., corresponding to reference block 568) can be represented as w2. In some cases (e.g., P1 is in bounds and P0 is out of bounds), w1 is equal to 0 and w2 is equal to 64 (or 32), which means that only P1 (e.g., reference block 556) is used as a composite predictor. In some cases (e.g., P1 is in bounds and P0 is out of bounds), w1 is equal to 1 and w2 is equal to 63 (or 31), which means that P1 (e.g., reference block 556) is more important; however, the padding value provided by P0 (e.g., reference block 568) is still used.

[0097] In some embodiments, multiple fixed weight factor values ​​are predefined and the index of the best factor value (e.g., the lowest cost factor value and / or the factor value that results in the most accurate reconstruction) is signaled in the high-level syntax (e.g., in a picture or tile header).

[0098] In some embodiments, the weight factors for P0 and P1 depend on the distance between the two reference frames and the current frame. In some embodiments, if the distance between the reference picture of P0 and the current frame is less than the distance between the reference frame of P1 and the current frame, a higher weight factor is assigned to P0. In some embodiments, multiple predefined weight factor pairs are stored in a lookup table. In some embodiments, the decision of which weight factor pair to use is implicitly derived based on the distance between the two reference frames and the current frame and / or whether the current region is outside the corresponding reference picture of P0 or P1.

[0099] In some embodiments, for P0 (e.g., reference boundary 566) or P1 (e.g., reference boundary 568), the weight factor gradually increases / decreases from the reference boundary (or the boundary in the reference picture). For example, for P0, the first row starting from the boundary uses the weight factor w0, the second row uses the weight factor w0+d, and the third row uses w0+2d, while the corresponding first row in P1 uses 64-w0, the second row uses 64-w0-d, and the third row uses 64-w0-2d.

[0100] In some embodiments, a wedge mask generation formula (e.g., Formula 2) is used to generate a weight factor to calculate the distance between the sample position (e.g., within region 562) and the reference boundary in P0 (or P1). Depending on the blending function, a lookup table and / or a clamp function (e.g., Formula 4) may be used to calculate the corresponding weight factor.

[0101] In some embodiments, a nonlinear mixing function is used, such as a sigmoid function, a hyperbolic tangent function, a trigonometric function, an exponential-based function, or a polynomial function.

[0102] In some embodiments, for P0 / P1, the areas within the boundary use the same weight (eg, to reduce complexity). For example, the weight of P0 may be 1, while the weight of P1 may be 63.

[0103] (A1) In one aspect, some embodiments include a method of video encoding (e.g., method 600). In some embodiments, the method is performed on a computing system (e.g., server system 112) having a memory and a control circuit. In some embodiments, the method is performed on a codec module (e.g., codec module 320). In some embodiments, the method is performed on a source encoding component (e.g., source encoder 202), an encoding engine (e.g., encoding engine 212), and / or an entropy encoder (e.g., entropy encoder 214). The method includes: (i) receiving a current picture and a current block within the current picture, wherein the current block is encoded using information from a first reference block and a second reference block (e.g., a composite prediction mode); (ii) determining that at least one of the first reference block and the second reference block is at least partially located outside a corresponding reference boundary (e.g., reference boundary 566 or reference boundary 558); and (iii) encoding the portion of the current block by averaging a reference value of the first reference block and a reference value of the second reference block based on determining that the portion of the current block corresponds to a first region, wherein the first region (a) is located between a corresponding reference boundary of the first reference block and a corresponding reference boundary of the second reference block. The method further comprises: (a) encoding the portion of the current block according to the first reference block and the second reference block, wherein the portion of the current block corresponds to a second region (e.g., region 562) and the second region is located only within one of the corresponding reference boundary of the first reference block and the corresponding reference boundary of the second reference block: (a) deriving a weighted reference value by applying corresponding weights to a reference value of the first reference block and a reference value of the second reference block; and (b) encoding the portion by combining the weighted reference value of the first reference block and the weighted reference value of the second reference block. For example, the corresponding reference boundary is a picture boundary, a slice boundary, a sub-picture boundary, or a tile boundary.

[0104] (A2) In some embodiments of A1, the method further includes: sending the encoded part via a video code stream.

[0105] (A3) In some embodiments of A1 or A2, the corresponding weight is a first fixed weight set among a plurality of fixed weight sets, and the first fixed weight set is identified based on a measured accuracy or an estimated accuracy of each fixed weight set.

[0106] (A4) In some embodiments of any one of A1 to A3, encoding the portion by averaging the reference value of the first reference block and the reference value of the second reference block includes: (i) deriving a second weighted reference value by applying a corresponding second weight to the reference value of the first reference block and the reference value of the second reference block, wherein the corresponding second weight is based on the distance between the first and second reference blocks and the current block; and (ii) encoding the portion by combining the second weighted reference value.

[0107] (A5) In some embodiments of A4, the first reference block is closer to the current block than the second reference block, a higher relative weight is assigned to the reference value of the first reference block, and a lower relative weight is assigned to the reference value of the second reference block.

[0108] (A6) In some embodiments of A4 or A5, the corresponding second weight is a first fixed weight set among multiple fixed weight sets, and the first fixed weight set is identified based on a measured accuracy or an estimated accuracy of each fixed weight set.

[0109] (A7) In some embodiments of any one of A1-A6, the corresponding weight is identified based on a distance between the second area and a corresponding reference boundary.

[0110] (A8) In some embodiments of any one of A1-A7, the corresponding weights are generated using one or more wedge mask generation formulas.

[0111] (A9) In some embodiments of any one of A1-A8, the first region and the second region are mixed using a nonlinear mixing function, and the nonlinear mixing function is used to determine a weight factor based on the distance from the boundary between the first region and the second region.

[0112] (A10) In some embodiments of any one of A1-A9, the corresponding weights are fixed. For example, the corresponding weight of the first reference block is assigned to 0, and the corresponding weight of the second reference block is assigned to n, where n is the maximum available weight. As another example, the corresponding weight of the first reference block is assigned to 1, and the corresponding weight of the second reference block is assigned to n-1, where n is the maximum available weight.

[0113] (A11) In some embodiments of any one of A1-A3 and A7-A10, encoding the portion by averaging the reference value of the first reference block and the reference value of the second reference block includes: averaging the unweighted reference value of the first reference block and the unweighted reference value of the second reference block.

[0114] (B1) On the other hand, some embodiments include a method of video decoding. In some embodiments, the method is performed on a computing system (e.g., server system 112) having a memory and a control circuit. In some embodiments, the method is performed on a decoding module (e.g., codec module 320). In some embodiments, the method is performed on a parser (e.g., parser 254), a motion prediction component (e.g., motion compensation prediction unit 260), and / or an intra-frame prediction component (e.g., intra-frame picture prediction unit 262). The method includes: (i) receiving a video code stream (e.g., an encoded video sequence), the video code stream including a current picture (e.g., current picture 502) and a current block in the current picture, wherein the current block is encoded using information from a first reference block (e.g., reference block 514) and a second reference block (e.g., reference block 512); (ii) determining that at least one of the first reference block and the second reference block is at least partially located outside a corresponding reference boundary; (iii) based on determining that the current block portion corresponds to a first region, reconstructing the portion by averaging a reference value of the first reference block and a reference value of the second reference block, wherein the first region: (a) (a) deriving a weighted reference value by applying a corresponding weight to a reference value of the first reference block and a reference value of the second reference block; and (b) reconstructing the portion by combining the weighted reference value of the first reference block and the weighted reference value of the second reference block. In some embodiments, receiving the video bitstream includes: receiving a syntax element indicating that a hybrid mode is to be used to predict the current block. For example, the first region is reconstructed by using a simple average, and the second region is reconstructed by using an implicit mask weighting. In some embodiments, the corresponding weight is not signaled in the video bitstream (e.g., derived at a decoder component). In some embodiments, determining that the current block is encoded using information from a first reference block and a second reference block includes determining to use a compound prediction mode (eg, a compound inter prediction mode) to predict the current block.

[0115] (B2) In some embodiments of B1, the corresponding weight is a first fixed weight set in a plurality of fixed weight sets. The first fixed weight set is identified according to a second syntax element in the video bitstream. For example, a plurality of fixed weight factor values ​​are predefined, and an index of an optimal factor value (e.g., determined by an encoder component) is signaled in a high-level syntax (e.g., a picture or tile header).

[0116] (B3) In some embodiments of B1 or B2, reconstructing the portion by averaging the reference value of the first reference block and the reference value of the second reference block includes: (i) deriving a second weighted reference value by applying a corresponding second weight to the reference value of the first reference block and the reference value of the second reference block, wherein the corresponding second weight is based on the distance between the first and second reference blocks and the current block; and (ii) reconstructing the portion by combining the second weighted reference value. For example, the weights assigned to P0 (e.g., reference block 556) and P1 (e.g., reference block 558) in the first region depend on the distance between the two reference frames and the current frame. In some embodiments, the corresponding second weight is fixed (e.g., the corresponding second weight is the same in all regions within the corresponding reference boundary).

[0117] (B4) In some embodiments of B3, the first reference block is closer to the current block than the second reference block, a higher relative weight is assigned to the reference value of the first reference block, and a lower relative weight is assigned to the reference value of the second reference block. For example, if the distance between the reference picture of P0 and the current picture is smaller than the distance between the reference picture of P1 and the current picture, a higher weight factor is assigned to P0.

[0118] (B5) In some embodiments of B3 or B4, the corresponding second weight is a first fixed weight set in a plurality of fixed weight sets, and the first fixed weight set is identified based on a syntax element in the video bitstream. For example, a plurality of predefined weight factor pairs are stored in a lookup table, and a decision of which weight factor pair to use is implicitly derived based on a distance between two reference frames and the current frame and / or whether the portion corresponds to an area outside a reference boundary in P0 or P1.

[0119] (B6) In some embodiments of any one of B1-B5, the corresponding weight is identified based on the distance between the second area and the corresponding reference boundary. In some embodiments, starting from the reference boundary in P0 (or P1) (or the boundary in the reference picture), the weight factor gradually increases / decreases. For example, in P0, the first row outside the reference boundary uses the first weight factor w0, the second row outside the reference boundary uses the second weight factor w0+d, the third row outside the reference boundary uses the third weight factor w0+2d, and so on. In this example, in P1, the corresponding first row can use 64-w0, the second row can use 64-w0-d, and the third row can use 64-w0-2d.

[0120] (B7) In some embodiments of any one of B1-B6, the corresponding weights are generated using one or more wedge mask generation formulas. For example, a wedge mask generation formula is used to generate weight factors to calculate the distance between the sample position in the first region and the reference boundary in P0 (or P1). In this example, a lookup table or a clamp function is used to calculate the corresponding weight factors according to the blending function.

[0121] (B8) In some embodiments of any one of B1-B7, a nonlinear mixing function is used to mix the first region and the second region, and the nonlinear mixing function is used to determine a weight factor based on a distance from a boundary between the first region and the second region. For example, the mixing function is a nonlinear mixing function, such as a sigmoid function, a hyperbolic tangent function, a trigonometric function, an exponential-based function, or a polynomial function.

[0122] (B9) In some embodiments of any one of B1-B8, the corresponding weights are fixed. In some embodiments, the sum of the corresponding weights is equal to 2 to the power of n. As an example, for areas outside the corresponding reference boundary, the weight factor of the first reference block is set to be equal to w1, and, for areas outside the corresponding reference boundary, the weight factor of the second reference block is set to be equal to w2. In some embodiments, the same weight is used for all areas that are only within one of the corresponding reference boundaries of the first reference block and the corresponding reference boundary of the second reference block. For example, the prediction block from P0 uses weight 1 and the prediction block from P1 uses weight 63.

[0123] (B10) In some embodiments of any one of B1-B9, the corresponding weight of the first reference block is assigned to 0, and the corresponding weight of the second reference block is assigned to n, where n is the maximum available weight. For example, for a given part, w1 of P0 is equal to 0, and w2 of P1 is equal to 64 (or 32), which means that only P1 contributes to the composite predictor.

[0124] (B11) In some embodiments of any one of B1-B9, the corresponding weight of the first reference block is assigned to 1, and the corresponding weight of the second reference block is assigned to n-1, where n is the maximum available weight. For example, for a given part, w1 of P0 is equal to 1, and w2 of P1 is equal to 63 (or 31), which means that P1 is more important, but P0 is still used (e.g., P0 corresponds to a fill value).

[0125] (B12) In some embodiments of any one of B1-B2 and B6-B11, reconstructing the part by averaging the reference value of the first reference block and the reference value of the second reference block includes: averaging the unweighted reference value of the first reference block and the unweighted reference value of the second reference block.

[0126] (B13) In some embodiments of any one of B1-B12, the corresponding reference boundary is a picture boundary, a slice boundary, a sub-picture boundary, or a tile boundary.

[0127] On the other hand, some embodiments include a computing system (e.g., server system 112), the computing system including a control circuit (e.g., control circuit 302) and a memory (e.g., memory 314) coupled to the control circuit. The memory stores one or more instruction sets configured to be executed by the control circuit, the one or more instruction sets including: instructions for executing any method described in the present disclosure (e.g., A1-A11 and B1-B13 described above).

[0128] On the other hand, some embodiments include a non-transitory computer-readable storage medium for executing one or more instruction sets by control circuitry of a computing system, the one or more instruction sets including instructions for performing any of the methods described in the present disclosure (e.g., A1-A11 and B1-B13 described above).

[0129] It should be understood that although the terms "first", "second", etc. can be used to describe various elements in the present disclosure, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. The terms used in the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the claims. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "the" are intended to also include plural forms. It should also be understood that the terms "and / or" used in the present disclosure refer to any combination and all combinations including one or more related listed items. It should be understood that when used in this specification, the term "comprising" specifies the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or components.

[0130] As used herein, the term "if" may be interpreted to mean "when" or "upon" or "in response to determining" or "upon determining" or "in response to detecting" that the antecedent is true, depending on the context. Likewise, the phrase "if it is determined that [the antecedent is true]" or "if [the antecedent is true]" or "when [the antecedent is true]" may be interpreted to mean "upon determination" or "in response to determining" or "upon determining" or "upon detecting" or "in response to detecting" that the antecedent is true, depending on the context.

[0131] For purposes of explanation, the foregoing description has been described with reference to specific embodiments. However, the foregoing illustrative discussion is not intended to be exhaustive or to limit the claims to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments are chosen and described in order to best explain the principles of operation and practical application, thereby enabling others skilled in the art to implement the present disclosure.

Claims

1. A video decoding method, executed on a computing system having a memory and one or more processors, the method comprising: Receive a video bitstream, the video bitstream comprising a current picture, a current block within the current picture, and a syntax element indicating that a hybrid mode is used to predict the current block; Determining, based on syntax elements in the video bitstream, that the current block is encoded using information from a first reference block and a second reference block; determining that at least one of the first reference block and the second reference block is at least partially located outside a corresponding reference boundary; Reconstructing the portion of the current block by averaging a reference value of the first reference block and a reference value of the second reference block based on determining that the portion of the current block corresponds to a first region, the first region: (i) being located within a corresponding reference boundary of the first reference block and a corresponding reference boundary of the second reference block, or (ii) not being located within a corresponding reference boundary of the first reference block and a corresponding reference boundary of the second reference block; as well as According to determining that the portion of the current block corresponds to a second region, the second region is located only within one of a corresponding reference boundary of the first reference block and a corresponding reference boundary of the second reference block: deriving a weighted reference value by applying corresponding weights to the reference value of the first reference block and the reference value of the second reference block; as well as The portion is reconstructed by combining the weighted reference value of the first reference block and the weighted reference value of the second reference block.

2. The method according to claim 1, wherein: The corresponding weight is a first fixed weight set among a plurality of fixed weight sets, and wherein the first fixed weight set is identified according to a second syntax element in the video bitstream.

3. The method according to claim 1, wherein: The reconstructing the portion by averaging the reference value of the first reference block and the reference value of the second reference block comprises: deriving a second weighted reference value by applying corresponding second weights to the reference value of the first reference block and the reference value of the second reference block, wherein the corresponding second weights are based on distances between the first and second reference blocks and the current block; and The portion is reconstructed by combining the second weighted reference values.

4. The method according to claim 3, wherein: The first reference block is closer to the current block than the second reference block, and wherein a higher relative weight is assigned to a reference value of the first reference block and a lower relative weight is assigned to a reference value of the second reference block.

5. The method according to claim 3, wherein: The corresponding second weight is a first fixed weight set among a plurality of fixed weight sets, and wherein the first fixed weight set is identified according to a second syntax element in the video bitstream.

6. The method according to claim 1, wherein: The respective weights are identified based on a distance between the second region and the corresponding reference boundary.

7. The method according to claim 1, wherein: The corresponding weights are generated using one or more wedge mask generation formulas.

8. The method according to claim 1, wherein: The first region and the second region are blended using a nonlinear blending function for determining a weight factor based on a distance from a boundary between the first region and the second region.

9. The method according to claim 1, wherein: The corresponding weights are fixed.

10. The method according to claim 1, wherein: The corresponding weight of the first reference block is assigned to 0, and the corresponding weight of the second reference block is assigned to n, where n is the maximum available weight.

11. The method according to claim 1, wherein: The corresponding weight of the first reference block is assigned to 1, and the corresponding weight of the second reference block is assigned to n-1, where n is the maximum available weight.

12. The method according to claim 1, wherein: The reconstructing the portion by averaging the reference value of the first reference block and the reference value of the second reference block comprises averaging an unweighted reference value of the first reference block and an unweighted reference value of the second reference block.

13. The method according to claim 1, wherein: The corresponding reference boundary is a picture boundary, a slice boundary, a sub-picture boundary or a tile boundary.

14. A computing system comprising: Control circuit; Memory; as well as One or more sets of instructions stored in the memory and configured to be executed by the control circuit, the one or more sets of instructions comprising instructions for performing the following steps: Receive a video bitstream, the video bitstream comprising a current picture, a current block within the current picture, and a syntax element indicating that a hybrid mode is used to predict the current block; Determining, based on syntax elements in the video bitstream, that the current block is encoded using information from a first reference block and a second reference block; determining that at least one of the first reference block and the second reference block is at least partially located outside a corresponding reference boundary; Reconstructing the portion of the current block by averaging a reference value of the first reference block and a reference value of the second reference block based on determining that the portion of the current block corresponds to a first region, the first region: (i) being located within a corresponding reference boundary of the first reference block and a corresponding reference boundary of the second reference block, or (ii) not being located within a corresponding reference boundary of the first reference block and a corresponding reference boundary of the second reference block; as well as According to determining that the portion of the current block corresponds to a second region, the second region is located only within one of a corresponding reference boundary of the first reference block and a corresponding reference boundary of the second reference block: deriving a weighted reference value by applying corresponding weights to the reference value of the first reference block and the reference value of the second reference block; as well as The portion is reconstructed by combining the weighted reference value of the first reference block and the weighted reference value of the second reference block.

15. The computing system of claim 14, wherein: The corresponding weight is a first fixed weight set among a plurality of fixed weight sets, and wherein the first fixed weight set is identified according to a second syntax element in the video bitstream.

16. The computing system of claim 14, wherein: The reconstructing the portion by averaging the reference value of the first reference block and the reference value of the second reference block comprises: deriving a second weighted reference value by applying corresponding second weights to the reference value of the first reference block and the reference value of the second reference block, wherein the corresponding second weights are based on distances between the first and second reference blocks and the current block; and The portion is reconstructed by combining the second weighted reference values.

17. The computing system of claim 14, wherein: The respective weights are identified based on a distance between the second region and the corresponding reference boundary.

18. A non-transitory computer-readable storage medium storing one or more sets of instructions configured to be executed by a computing device having control circuitry and memory, the one or more sets of instructions comprising instructions for performing the following steps: Receive a video bitstream, the video bitstream comprising a current picture, a current block within the current picture, and a syntax element indicating that a hybrid mode is used to predict the current block; Determining, based on syntax elements in the video bitstream, that the current block is encoded using information from a first reference block and a second reference block; determining that at least one of the first reference block and the second reference block is at least partially located outside a corresponding reference boundary; Reconstructing the portion of the current block by averaging a reference value of the first reference block and a reference value of the second reference block based on determining that the portion of the current block corresponds to a first region, the first region: (i) being located within a corresponding reference boundary of the first reference block and a corresponding reference boundary of the second reference block, or (ii) not being located within a corresponding reference boundary of the first reference block and a corresponding reference boundary of the second reference block; as well as According to determining that the portion of the current block corresponds to a second region, the second region is located only within one of a corresponding reference boundary of the first reference block and a corresponding reference boundary of the second reference block: deriving a weighted reference value by applying corresponding weights to the reference value of the first reference block and the reference value of the second reference block; as well as The portion is reconstructed by combining the weighted reference value of the first reference block and the weighted reference value of the second reference block.

19. The non-transitory computer-readable storage medium of claim 18, wherein: The corresponding weight is a first fixed weight set among a plurality of fixed weight sets, and wherein the first fixed weight set is identified according to a second syntax element in the video bitstream.

20. The non-transitory computer-readable storage medium of claim 18, wherein: The reconstructing the portion by averaging the reference value of the first reference block and the reference value of the second reference block comprises: deriving a second weighted reference value by applying corresponding second weights to the reference value of the first reference block and the reference value of the second reference block, wherein the corresponding second weights are based on distances between the first and second reference blocks and the current block; and The portion is reconstructed by combining the second weighted reference values.

Images