Apparatus and method for decoding video data

Abstract

A method for decoding video data by an electronic device is provided. The electronic device receives video data and determines a block unit from an image frame according to the video data. The electronic device further determines a plurality of candidate positions including at least one non-adjacent position not adjacent to the block unit, and determines a plurality of motion candidates from the candidate positions. The electronic device further selects a plurality of collocated blocks each included in one of one or more collocated frames (collocated blocks) included in the video data based on the motion candidates, and determines a first cost value for each of the collocated blocks. Then, the electronic device determines at least one sub-block-based temporal motion vector prediction (SbTMVP) candidate based on the motion candidates and the first cost value, and reconstructs the block unit based on the at least one SbTMVP candidate.

Description

Detailed Description of the Invention

[0001] [Technical field] This disclosure claims the benefit of and priority to U.S. Provisional Patent Application Serial No. 63 / 265442, filed December 15, 2021, entitled "TEMPORAL-BASED PREDICTION TOOLS," and U.S. Provisional Patent Application Serial No. 63 / 265777, filed December 20, 2021, entitled "SUBBLOCK TMVP IMPROVEMENT BASED ON TEMPLATE MATCHING," the contents of which are incorporated by reference in their entireties into this disclosure.

[0002] TECHNICAL FIELD This disclosure relates generally to video decoding, and more specifically, to techniques using template prediction.

[0003] [Background technology] Sub-block-based temporal motion vector prediction (SbTMVP) is a coding tool for video coding. In a conventional video coding method, an encoder and a decoder can determine a collocated block in a collocated frame for a candidate block in an image frame, and generate one of multiple SbTMVP candidates for predicting the candidate block.

[0004] The encoder and decoder may divide the collocated block into a plurality of collocated sub-blocks and determine a plurality of first sub-block vectors Vs1 for the collocated sub-blocks in the collocated block using the motion information of the collocated block. The encoder and decoder may then divide the candidate block into a plurality of target sub-blocks and derive a plurality of second sub-block vectors Vs2 for the target sub-blocks in the block unit based on the first sub-block vectors Vs1 of the collocated sub-blocks. Each of the second sub-block vectors Vs2 may be derived based on a corresponding one of the first sub-block vectors Vs1, a reference distance between the image frame and the reference frame of the block unit, and a collocated distance between the collocated frame and the collocated reference frame of the collocated block.

[0005] However, since the SbTMVP candidates of a candidate block are only arranged based on the collection order, when predicting a candidate block using the SbTMVP candidates, the coding efficiency for signaling the index for determining the selected SbTMVP candidate may not be high enough. Therefore, the encoder and decoder may need a new SbTMVP to predict or reconstruct the candidate block more accurately. 〔overview〕

[0006] The present disclosure is directed to an apparatus and method for predicting block units in an image frame using template prediction.

[0007] In a first aspect of the present disclosure, a method for decoding video data and an electronic device for performing the method are provided, the method includes receiving video data, determining a block unit from an image frame according to the video data, determining a plurality of candidate positions including at least one non-adjacent position not adjacent to the block unit, determining a plurality of motion candidates from the plurality of candidate positions, selecting a plurality of collocated blocks each included in one of one or more collocated frames included in the video data based on the plurality of motion candidates, determining a first cost value for each of the plurality of collocated blocks, determining at least one sub-block-based temporal motion vector prediction (SbTMVP) candidate based on the plurality of motion candidates and the plurality of first cost values, and reconstructing a block unit based on the at least one SbTMVP candidate.

[0008] In another embodiment of the first aspect, the plurality of candidate locations further includes at least one of a plurality of adjacent locations adjacent to the block unit.

[0009] An embodiment of the first aspect further includes the steps of: determining, from the image frame, a plurality of neighboring blocks adjacent to the block unit based on the plurality of candidate positions, where at least one of the plurality of neighboring blocks covers one of the at least one non-adjacent positions; determining, from the image frame, a template region adjacent to the block unit and a plurality of neighboring regions each adjacent to one of the plurality of neighboring blocks; determining a second cost value based on the template region and each of the plurality of neighboring regions; selecting a plurality of candidate blocks based on the plurality of second cost values; and determining the plurality of motion candidates from the plurality of candidate blocks.

[0010] In another embodiment of the first aspect, each of the multiple motion candidates indicates a motion vector, list information, and frame information of a candidate block covering one of the multiple candidate positions, and if a frame index in the frame information of a particular candidate block is equal to one of at least one predetermined value, a reference frame of a particular one of the candidate blocks selected from the reference list indicated by the list information of the particular candidate block is included in one or more collocated frames.

[0011] In another embodiment of the first aspect, one of the plurality of collocated blocks is selected from a reference frame for a particular candidate block based on the motion vector and a corresponding one of the plurality of candidate locations.

[0012] In another embodiment of the first aspect, if a frame index of the frame information of the particular candidate block is different from at least one predetermined value, the reference frame of the particular candidate block is excluded from the one or more collocated frames.

[0013] In another embodiment of the first aspect, each of the multiple motion candidates indicates a motion vector, list information, and frame information of a candidate block covering one of the multiple candidate positions, and at least one reference frame for a particular one of the candidate blocks is determined from two reference lists for the particular candidate block if the particular candidate block is a bi-predictive block, and the at least one reference frame is included in the one or more collocated frames if one of two frame indexes in the frame information of the particular candidate block is equal to one of at least one predetermined value.

[0014] An embodiment of the first aspect further includes the steps of adding the at least one SbTMVP candidate to a candidate list, selecting a prediction candidate from the candidate list based on a prediction index, and reconstructing the block unit based on the prediction mode.

[0015] In another embodiment of the first aspect, the plurality of collocated blocks is selected from a plurality of collocated frames.

[0016] An embodiment of the first aspect further includes the steps of determining a template region adjacent to the block unit from the image frame, determining a plurality of collocated regions from one or more collocated frames, each adjacent to one of a plurality of collocated blocks, and determining the first cost value between the template region and each of the plurality of collocated regions.

[0017] BRIEF DESCRIPTION OF THE DRAWINGS Aspects of the present disclosure are best understood from the following detailed disclosure and corresponding figures, in which: Various features are not drawn to scale and dimensions of various features may be arbitrarily increased or decreased for clarity of discussion. FIG. 1 is a block diagram of a system configured to encode and decode video data according to an embodiment of the present disclosure. FIG. 2 is a block diagram of a decoder module of the second electronic device illustrated in FIG. 1 according to an embodiment of the present disclosure. FIG. 3 shows a flowchart of a method for decoding video data by an electronic device according to an embodiment of the present disclosure. FIG. 4 is an explanatory diagram illustrating an exemplary embodiment of a block and a plurality of candidate positions according to an embodiment of the present disclosure. 5A, 5B, and 5C are diagrams illustrating an example implementation of a template region and different neighboring regions according to one embodiment of the present disclosure. FIG. 6 is an explanatory diagram showing an example embodiment of an image frame having a block unit and a template region, and a collocated frame having a collocated block and a collocated region according to an embodiment of the present disclosure. FIG. 7 is an explanatory diagram showing an example implementation of an image frame having a block unit and a template region, and a collocated frame having a collocated block, a shift block, and a collocated region according to one embodiment of the present disclosure. FIG. 8 shows a flowchart of a method for decoding video data by an electronic device according to an embodiment of the present disclosure. 9A and 9B are diagrams illustrating an example embodiment of a template region and a reference region according to an embodiment of the present disclosure. FIG. 10 is a block diagram of an encoder module of the first electronic device shown in FIG. 1 according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] The following disclosure contains specific information relevant to the implementation of the present disclosure. The figures and corresponding detailed disclosure are directed to exemplary embodiments. However, the present disclosure is not limited to these exemplary embodiments. Other variations and exemplary embodiments of the present disclosure will occur to those skilled in the art.

[0019] Unless otherwise noted, like or corresponding elements in the figures may be indicated with like or corresponding reference designators. The figures and illustrations in this disclosure are generally not to scale and are not intended to correspond to actual relative dimensions.

[0020] For purposes of consistency and ease of understanding, like features are identified in the exemplary figures by reference designators (although not shown in some exemplary embodiments), however, features in different embodiments may differ in other respects and are not intended to be narrowly limited to those illustrated in the figures.

[0021] In this disclosure, the phrases "in one embodiment" or "in some embodiments" may refer to one or more of the same or different embodiments. The term "coupled" means connected through an intervening component, whether directly or indirectly, and not necessarily limited to a physical connection. The term "comprising" means "including, but not necessarily limited to," and specifically indicates an open-ended inclusion or membership in such described combinations, groups, series, and the like.

[0022] For purposes of explanation and not limitation, specific details such as functional entities, techniques, protocols, and standards are set forth to provide an understanding of the disclosed technology. Detailed disclosures of well-known methods, techniques, systems, and architectures are omitted so as not to obscure the present disclosure with unnecessary detail.

[0023] Those skilled in the art will recognize that the disclosed code function(s) or algorithm(s) described in this disclosure may be implemented by hardware, software, or a combination of software and hardware. The disclosed functions may correspond to modules that are software, hardware, firmware, or a combination thereof.

[0024] A software implementation may include a program having computer-executable instructions stored on a computer-readable medium, such as a memory or other type of storage device. For example, one or more microprocessors or general-purpose computers having communication processing capabilities can be programmed with the executable instructions to perform the disclosed function(s) or algorithm(s).

[0025] A microprocessor or general-purpose computer can be formed using an application specific integrated circuit (ASIC), a programmable logic array, and / or one or more digital signal processors (DSPs). Although some of the disclosed implementations are directed to software installed and executed on computer hardware, alternative implementations implemented as firmware, or as hardware, or as a combination of hardware and software are within the scope of this disclosure. Computer-readable media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, compact disc read-only memory (CD ROM), magnetic cassette, magnetic tape, magnetic disk storage, or any other equivalent medium capable of storing computer-readable instructions.

[0026] 1 shows a block diagram of a system 100 configured to encode and decode video data according to an embodiment of the present disclosure. The system 100 includes a first electronic device 110, a second electronic device 120, and a communication medium 130.

[0027] First electronic device 110 may be a source device including any device configured to encode video data and transmit the encoded video data to communication medium 130. Second electronic device 120 may be a destination device including any device configured to receive the encoded video data via communication medium 130 and decode the encoded video data.

[0028] The first electronic device 110 can communicate with the second electronic device 120 via a communication medium 130 in a wired or wireless manner. The first electronic device 110 may include a source module 112, an encoder module 114, and a first interface 116. The second electronic device 120 may include a display module 122, a decoder module 124, and a second interface 126. The first electronic device 110 may be a video encoder, and the second electronic device 120 may be a video decoder.

[0029] The first electronic device 110 and / or the second electronic device 120 may be a mobile phone, tablet, desktop, notebook, or other electronic device. Figure 1 illustrates an example of the first electronic device 110 and the second electronic device 120. The first electronic device 110 and the second electronic device 120 may include more or fewer components than shown or may have a different configuration of the various components shown.

[0030] The source module 112 may include a video capture device to capture new video, a video archive to store previously captured video, and / or a video feed interface to receive video from a video content provider. The source module 112 may generate computer graphics-based data as the source video, or may generate a combination of live video, archive video, and computer-generated video as the source video. The video capture device may be a charge-coupled device (CCD) image sensor, a complementary metal-oxide semiconductor (CMOS) image sensor, or a camera.

[0031] The encoder module 114 and the decoder module 124 may each be implemented as any of a variety of suitable encoder / decoder circuits, such as one or more microprocessors, central processing units (CPUs), graphic processing units (GPUs), systems on chips (SoCs), digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. If implemented partially in software, the device may store a program having instructions for the software on a suitable non-transitory computer readable medium and execute the instructions in hardware using one or more processors to perform the disclosed methods. Each of the encoder module 114 and the decoder module 124 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder / decoder (CODEC) within the device.

[0032] The first interface 116 and the second interface 126 may follow a customized protocol or an existing or de facto standard including, but not limited to, Ethernet, IEEE 802.11 or IEEE 802.15 series, Wireless USB, or a telecommunications standard including, but not limited to, Global System for Mobile Communications (GSM), Code-Division Multiple Access 2000 (CDMA2000), Time Division Synchronous Code Division Multiple Access (TD-SCDMA), Worldwide Interoperability for Microwave Access (WiMAX), Third Generation Partnership Project Long-Term Evolution (3GPP-LTE), or Time-Division LTE (TD-LTE). The first interface 116 and the second interface 126 may each include any device configured to transmit and / or store a compliant video bitstream over a communication medium 130 and receive a compliant video bitstream over a communication medium 130.

[0033] The first interface 116 and the second interface 126 may include computer system interfaces that allow compliant video bitstreams to be stored on or received from a storage device. For example, the first interface 116 and the second interface 126 may include chipsets that support Peripheral Component Interconnect (PCI) and Peripheral Component Interconnect Express (PCIe) bus protocols, proprietary bus protocols, Universal Serial Bus (USB) protocols, Inter-Integrated Circuit (I2C) protocols, or any other logical and physical structures that may be used to interconnect peer devices.

[0034] The display module 122 may include a display using liquid crystal display (LCD) technology, plasma display technology, organic light emitting diode (OLED) display technology, or light emitting polymer display (LPD) technology, as well as other display technologies used in other implementations. The display module 122 may include a high definition display or an ultra high definition display.

[0035] 2 is a block diagram of the decoder module 124 of the second electronic device 120 shown in FIG. 1 according to an embodiment of the present disclosure. The decoder module 124 includes an entropy decoder (e.g., entropy decode unit 2241), a prediction processor (e.g., prediction process unit 2242), an inverse quantization / inverse transform processor (e.g., inverse quantization / inverse transform unit 2243), an adder (e.g., adder 2244), a filter (e.g., filtering unit 2245), and a decoded picture buffer (e.g., decoded picture buffer 2246). The prediction process unit 2242 further includes an intra-prediction process unit (e.g., intra-prediction unit 22421) and an inter-prediction process unit (e.g., inter-prediction unit 22422). The decoder module 124 receives a bitstream, decodes the bitstream, and outputs a decoded video.

[0036] 1, and may perform a parsing operation on the bitstream to extract syntax elements from the bitstream. As part of the parsing operation, the entropy decoding unit 2241 may entropy decode the bitstream to generate quantized transform coefficients, quantization parameters, transform data, motion vectors, intra modes, partition information, and other syntax information.

[0037] The entropy decoding unit 2241 may perform context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding technique to generate quantized transform coefficients. The entropy decoding unit 2241 may provide the quantized transform coefficients, quantization parameters, and transform data to the inverse quantization / inverse transform unit 2243 and provide motion vectors, intra modes, partition information, and other syntax information to the prediction process unit 2242.

[0038] The prediction process unit 2242 may receive syntax elements such as motion vectors, intra modes, partition information, and other syntax information from the entropy decoding unit 2241. The prediction process unit 2242 may receive syntax elements including the partition information and divide the image frame according to the partition information.

[0039] Each of the image frames may be divided into at least one image block according to the partition information. The at least one image block may include a luminance block for reconstructing a plurality of luminance samples and at least one chrominance block for reconstructing a plurality of chrominance samples. The luminance block and the at least one chrominance block may be further divided to generate a macroblock, a coding tree unit (CTU), a coding block (CB), a subdivision thereof, and / or another equivalent coding unit.

[0040] During the decoding process, the prediction process unit 2242 may receive prediction data including intra mode or motion vectors for a particular one of the current image blocks of the image frame, which may be one of the luminance or chrominance blocks of the particular image frame.

[0041] The intra prediction unit 22421 may perform intra prediction encoding of the current block unit relative to one or more neighboring blocks in the same frame as the current block unit based on a syntax element associated with an intra mode to generate a predicted block. The intra mode may specify the location of a reference sample selected from the neighboring block in the current frame. The intra prediction unit 22421 may reconstruct multiple chroma components of the current block unit based on multiple luma components of the current block unit when the chroma components are reconstructed by the prediction process unit 2242.

[0042] The intra prediction unit 22421 may reconstruct multiple chroma components of the current block unit based on the multiple luma components of the current block unit when the luma component of the current block is reconstructed by the prediction process unit 2242.

[0043] The inter prediction unit 22422 may perform relative inter prediction encoding of the current block unit relative to one or more blocks in one or more reference image blocks based on a syntax element related to a motion vector to generate a predicted block. The motion vector may indicate the displacement of the current block unit in the current image block relative to the reference block unit in the reference image block. The reference block unit is a block determined to closely match the current block unit. The inter prediction unit 22422 may receive the reference image blocks stored in the decoded picture buffer 2246 and reconstruct the current block unit based on the received reference image blocks.

[0044] The inverse quantization / inverse transform unit 2243 may apply inverse quantization and an inverse transform to reconstruct the residual block in the pixel domain. The inverse quantization / inverse transform unit 2243 may apply inverse quantization to the residual quantized transform coefficients to generate residual transform coefficients, and apply an inverse transform to the residual transform coefficients to generate the residual block in the pixel domain.

[0045] The inverse transform may be applied by a transformation process such as the discrete cosine transform (DCT), discrete sine transform (DST), adaptive multi-layer transform (AMT), mode-dependent non-separable quadratic transform (MDNSST), hypercube-Givens transform (HyGT), signal-dependent transform, Karhunen-Loewe transform (KLT), wavelet transform, integer transform, subband transform, or a conceptually similar transform. The inverse transform can return residual information from a transform domain, such as the frequency domain, back to the pixel domain. The degree of inverse quantization can be changed by adjusting the quantization parameter.

[0046] The adder 2244 adds the reconstructed residual block to the prediction block provided by the prediction process unit 2242 to generate a reconstructed block.

[0047] The filtering unit 2245 may include a deblocking filter, a sample adaptive offset (SAO) filter, a bilateral filter, and / or an adaptive loop filter (ALF) to remove blocking artifacts from the reconstructed blocks. In addition to the deblocking filter, the SAO filter, the bilateral filter, and the ALF, additional filters (in-loop or post-loop) may also be used. Such filters may filter the output of the summer 2244, although they are not explicitly illustrated for simplicity. After the filtering unit 2245 performs the filtering process on the reconstructed blocks of a particular image frame, the filtering unit 2245 may output the decoded video to the display module 122 or other video receiving unit.

[0048] The decoded picture buffer 2246 may be a reference picture memory that stores reference blocks used by the prediction process unit 2242 in decoding the bitstream (in inter-coding mode). The decoded picture buffer 2246 may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The decoded picture buffer 2246 may be on-chip with other components of the decoder module 124 or off-chip relative to those components.

[0049] 3 shows a flowchart of a method 300 for decoding video data by an electronic device according to an embodiment of the present disclosure. The method 300 is merely exemplary, as there are various ways to perform the decoding of video data.

[0050] Method 300 may be performed using the configurations illustrated in Figures 1 and 2, the various elements of which are referenced with respect to method 300. Each block illustrated in Figure 3 may represent one or more processes, methods, or subroutines that are performed.

[0051] The order of the blocks in Figure 3 is exemplary and may be changed: additional blocks may be added or fewer blocks may be utilized without departing from this disclosure.

[0052] At block 310, the decoder module 124 receives the video data. The video data received by the decoder module 124 may be a bitstream.

[0053] 1 and 2, the second electronic device 120 may receive a bitstream from an encoder, such as the first electronic device 110, or another video provider, via a second interface 126. The second interface 126 may provide the bitstream to a decoder module 124.

[0054] The entropy decoding unit 2241 may decode the bitstream to determine prediction indications and partition indications for the image frames. The decoder module 124 may then further reconstruct the image frames based on the prediction indications and partition indications. The prediction indications and partition indications may include flags and indexes.

[0055] In block 320, the decoder module 124 determines block units from the image frame according to the video data.

[0056] 1 and 2, the decoder module 124 may determine an image frame based on a bitstream, and divide the image frame to determine block units according to a partition instruction in the bitstream. For example, the decoder module 124 may divide the image frame to generate a plurality of CTUs, and further divide one of the CTUs to determine block units according to a partition instruction based on any video encoding standard.

[0057] In block 330, the decoder module 124 determines a number of candidate locations, including at least one non-adjacent location that is not adjacent to the block unit.

[0058] The candidate locations may be a plurality of proximate locations adjacent to the block unit. The proximate locations may be selected from at least one of a plurality of proximate locations adjacent to the block unit or a plurality of non-proximate locations not adjacent to the block unit. Thus, in some embodiments, the candidate locations may include at least one of a proximate location or a non-proximate location. In some embodiments, the candidate locations may include at least one of a proximate location and at least one of a non-proximate location. In some embodiments, the candidate locations may include at least one of a non-proximate location and exclude a proximate location. In some embodiments, the candidate locations may exclude a non-proximate location and include at least one of a proximate location.

[0059] FIG. 4 is an illustration of an exemplary implementation of a block and multiple candidate positions according to an embodiment of the present disclosure. With reference to FIG. 1 and FIG. 2, the decoder module 124 may determine multiple adjacent positions 401-405 adjacent to a current block 400 and multiple non-adjacent positions 406-423 not adjacent to the block unit 400. Each of the adjacent positions 401-405 may be adjacent to one of the four corners of the block unit 400. Furthermore, the distance between the block unit 400 and one of the non-adjacent positions 406-423 may be based on one of the block height H, block width W, or diagonal length D of the block unit 400. For example, the horizontal distance and vertical distance between the top left position of the block unit 400 and the non-adjacent position 408 may be equal to W+1 and H+1 of the block unit 400, respectively, and the distance between the block unit 400 and the non-adjacent position 411 may be equal to a distance value generated by adding 1 to two times the block height H of the block unit 400.

[0060] In block 340, the decoder module 124 determines multiple motion candidates from the multiple candidate positions.

[0061] 1 and 2, the decoder module 124 may determine a plurality of neighboring blocks adjacent to the block unit based on the candidate positions. Each of the neighboring blocks may cover at least one of the candidate positions. Each of the candidate positions is located at the upper left position of a corresponding one of the neighboring blocks. Furthermore, the size of the neighboring blocks may be the same as the size of the block unit (i.e., the block height H, the block width W, or the diagonal length D).

[0062] The neighborhood block determined based on one of the adjacent locations may cover one or more of the adjacent locations, since some of the adjacent locations are adjacent to each other. The neighborhood block determined based on one of the non-adjacent locations may cover one of the non-adjacent locations. Furthermore, the neighborhood block determined based on one of the non-adjacent locations may cover only one of the non-adjacent locations, since the distance between the non-adjacent locations may be equal to or larger than the size of a block unit.

[0063] The decoder module 124 may determine a template region adjacent to the block unit from the image frame. The template region may include at least one of a plurality of adjacent regions. The first adjacent region may be an adjacent region located on the top side of the block unit, the second adjacent region may be an adjacent region located on the left side of the block unit, and the third adjacent region may be an adjacent region located on the top left side of the block unit. The adjacent regions may be reconstructed before reconstructing the block unit. The height of the first adjacent region may be equal to the number R of reconstructed samples of the first adjacent region along the vertical direction, and the width of the first adjacent region may be equal to the width of the block unit. The height of the second adjacent region may be equal to the height of the block unit, and the width of the second adjacent region may be equal to the number S of reconstructed samples of the second adjacent region along the horizontal direction. Furthermore, the height of the third adjacent region may be equal to the number R of reconstructed samples of the first adjacent region along the vertical direction, and the width of the third adjacent region may be equal to the number S of reconstructed samples of the second adjacent region along the horizontal direction. In one embodiment, the numbers R and S may be positive integers. Furthermore, the numbers R and S may be equal to or different from each other. Furthermore, the numbers R and S may be greater than or equal to 1. In some embodiments, the numbers R and S may be equal to 1.

[0064] The decoder module 124 may determine template regions adjacent to the block unit. The decoder module 124 may use the adjacent regions as template regions for determining the motion candidate. Furthermore, the decoder module 124 may use two of the adjacent regions as template regions for determining the motion candidate. For example, the decoder module 124 may use only the first and second adjacent regions as template regions for determining the motion candidate. Furthermore, the decoder module 124 may use only one of the adjacent regions as template regions for determining the motion candidate.

[0065] The decoder module 124 may determine a plurality of neighboring regions from the image frame. Each of the neighboring regions may be adjacent to a corresponding one of the neighboring blocks. Each of the neighboring regions may include at least one of the plurality of neighboring regions of the corresponding one of the neighboring blocks. The first neighboring region of a particular one of the neighboring blocks may be a neighboring region located on the upper side of the particular neighboring block, the second neighboring region of the particular neighboring block may be a neighboring region located on the left side of the particular neighboring block, and the third neighboring region of the particular neighboring block may be a neighboring region located on the upper left side of the particular neighboring block. The neighboring regions of the neighboring blocks may be reconstructed before reconstructing the block unit.

[0066] 5A, 5B, and 5C, there are illustrated diagrams of exemplary implementations of a template region and different neighborhood regions according to an embodiment of the present disclosure. FIG. 5A is an illustrative diagram of an exemplary implementation of a block unit 500, a neighborhood block 530 covering a non-adjacent position 508, and a neighborhood region 5300 according to an embodiment of the present disclosure. The size of the neighborhood block 530 may be the same as the size of the block unit 500. Since the neighborhood block 530 is located on the upper side and the left side of the block unit 500 and does not cover the block unit 500, the samples in the neighborhood block 530 may be reconstructed samples reconstructed before the block unit 500. Furthermore, the samples in the neighborhood region 5300 may also be reconstructed samples reconstructed before the block unit 500.

[0067] FIG. 5B is an illustration of an example embodiment of a block unit 500, a neighborhood block 540 covering a neighboring position 502, and a neighborhood region 5400, according to an embodiment of the present disclosure. The size of the neighborhood block 540 may be the same as the size of the block unit 500. However, since the neighborhood position 502 is located adjacent to the upper right corner of the block unit 500, some of the samples in the neighborhood block 540 may be reconstructed samples reconstructed before the block unit 500, and other samples in the neighborhood block 540 may be non-reconstructed samples. Thus, the decoder module 124 may determine a neighborhood region 5400 that is smaller than the template region 5000 of FIG. 5A to exclude the region covered by the block unit 500. Thus, since the neighborhood region 5400 is located above the block unit 500 and does not cover the block unit 500, the samples in the neighborhood region 5400 may also be reconstructed samples reconstructed before the block unit 500.

[0068] The decoder module 124 may determine a neighborhood cost value based on each of the template region and the multiple neighboring regions using a cost function to select a motion candidate for the block unit. In FIG. 5A, the decoder module 124 may directly compare the neighborhood region 5300 of the neighboring block 530 with the template region 5000 of the block unit 500 by the cost function. For example, the decoder module 124 may compare multiple neighborhood samples in the neighborhood region 5300 of the neighboring block 530 with the reconstructed samples in the template region 5000 of the block unit 500 to generate one of the neighborhood cost values. However, since the neighborhood region 5400 may be smaller than the template region 5000, the decoder module 124 may not directly compare the neighborhood region 5400 of the neighboring block 540 with the template region 5000 of the block unit 500 by the cost function. FIG. 5C is an illustration of an example embodiment of the block unit 500 and the secondary region 5001 according to an embodiment of the present disclosure. The secondary region 5001 may be determined based on the size and shape of the neighboring region 5400. The size and shape of the secondary region 5001 may be the same as the size and shape of the neighboring region 5400. Therefore, the decoder module 124 may directly compare the neighboring region 5400 of the neighboring block 540 with the template region 5001 of the block unit 500 by a cost function.

[0069] The cost function may be a template matching cost function, including but not limited to sum of absolute differences (SAD), sum of absolute transformed differences (SATD), mean absolute difference (MAD), mean squared difference (MSD), and structural similarity measure (SSIM). It should be noted that any cost function may be used without departing from this disclosure.

[0070] The decoder module 124 may select multiple candidate blocks based on the neighborhood cost values ​​calculated by the neighborhood regions. However, since the sizes of the neighborhood regions may be the same or different from each other, it may be unfair to directly compare the neighborhood cost values ​​with each other. The decoder module 124 may determine an averaged cost value for each of the neighborhood blocks based on the size of the neighborhood regions. For example, the averaged cost value of the neighborhood block 530 may be calculated by dividing the neighborhood cost value of the neighborhood region 5300 by 8, and the averaged cost value of the neighborhood block 540 may be calculated by dividing the neighborhood cost value of the neighborhood region 5400 by 5. In this manner, the decoder module 124 may compare the averaged cost values ​​with each other to select some of the neighborhood positions for determining the candidate block.

[0071] If the number of selected neighborhood positions is equal to 3, three neighborhood positions corresponding to three neighborhood blocks having three lowest averaged cost values ​​may be selected, and the motion candidates may include motion information of three candidate blocks covering the three neighborhood positions. If the number of selected neighborhood positions is equal to 1, one neighborhood position corresponding to one neighborhood block having the lowest averaged cost value may be selected, and the motion candidates may include motion information of one candidate block covering the one neighborhood position. Thus, if the number of selected neighborhood positions is equal to Y, Y neighborhood positions may be selected based on the Y lowest averaged cost values, and Y candidate blocks may be determined. The number Y may be a positive integer.

[0072] The candidate block may be a reconstructed block reconstructed before reconstructing the block unit. When one of the candidate blocks does not have motion information, the motion candidate may not be determined based on a corresponding one selected neighboring position. For example, when one of the candidate blocks is an intra-prediction block, one of the candidate blocks does not have motion information to be added to the motion candidate. In addition, the motion candidate may further include a history-based candidate determined from a first-in-first-out (FIFO) table that stores multiple previous previous motion information of multiple previous blocks reconstructed before reconstructing the block unit.

[0073] At block 350, the decoder module 124 selects multiple collocated blocks each included in one of the one or more collocated frames based on the multiple motion candidates.

[0074] Each of the multiple motion candidates may indicate a motion vector, list information, and frame information of a candidate block that covers one of the multiple candidate positions. Furthermore, the motion vector, list information, and frame information of a particular one of the candidate blocks may be used to predict the particular candidate block in order to reconstruct the particular candidate block.

[0075] The list information of the particular candidate block may include at least one of an inter prediction indication or a multiple list prediction flag. The inter prediction indication may indicate which of a first uni-prediction with a first reference list List0, a second uni-prediction with a second reference list List1, and a bi-prediction including a first reference list List0 and a second reference list List1 is used for the particular candidate block. Furthermore, the list prediction flag may include a first list prediction flag used to determine whether a first reference list List0 is used for the particular candidate block, and a second list prediction flag used to determine whether a second reference list List1 is used for the particular candidate block. In some embodiments, the list information of the particular candidate block may include both an inter prediction indication and a multiple list prediction flag for determining a reference list of the particular candidate block. Furthermore, the list information of the particular candidate block may include only an inter prediction indication for determining a reference list of the particular candidate block. Furthermore, the list information of the particular candidate block may include only a multiple list prediction flag for determining a reference list of the particular candidate block.

[0076] The frame information may include at least one frame index. When a particular candidate block is a bi-predictive block, the frame information of the particular candidate block may include two frame indexes to indicate two reference frames respectively included in the corresponding one of the reference lists. Furthermore, when a particular candidate block is a uni-predictive block, the frame information of the particular candidate block may include only one frame index to indicate one reference frame included in the corresponding one of the reference lists. In another embodiment, when a particular candidate block is a uni-predictive block, the frame information of the particular candidate block may also include two frame indexes. One of the two frame indexes may be used to indicate one reference frame included in the corresponding one of the reference lists, and the other of the two frame indexes may be equal to a predetermined frame value to indicate that the other one of the reference lists is not used for the particular candidate block.

[0077] 1 and 2, the decoder module 124 may determine one or more collocated frames based on at least one predetermined value. If one of the frame indexes in the frame information of the particular candidate block is equal to one of the at least one predetermined value, a reference frame of the particular candidate block selected from a corresponding one of the reference lists indicated by the list information of the particular candidate block may be included in one or more collocated frames. For example, the at least one predetermined value may include zero. In some embodiments, the particular candidate block may be a single predicted block predicted based on the second reference list List1. If the second frame index of the particular candidate block corresponding to the second reference list List1 is equal to zero, a reference frame of the particular candidate block selected from the second reference list List1 based on the second frame index may be included in one or more collocated frames. However, a reference frame of the particular candidate block selected from the second reference list List1 may be excluded from one or more collocated frames if the second frame index in the frame information of the particular candidate block is different from the at least one predetermined value. In this manner, the reference frame of a particular candidate block may be added to or excluded from one or more collocated frames based on the frame information.

[0078] In some embodiments, the particular candidate block may be a bi-predictive block predicted based on the first reference list List0 and the second reference list List1. Furthermore, some of the multiple reference candidates in the first reference list List0 may be identical to some of the multiple reference candidates in the second reference list List1. Thus, a first reference frame of the particular candidate block selected from the first reference list List0 based on the first frame index may be identical to a second reference frame of the particular candidate block selected from the second reference list List1 based on the second frame index. Thus, the number of reference frames of the particular candidate block determined from the two reference lists may be equal to 1 or 2 when the particular candidate block is a bi-predictive block.

[0079] If the first frame index of a particular candidate block corresponding to the first reference list List0 is equal to zero, the first reference frame of the particular candidate block selected from the first reference list List0 based on the first frame index may be included in one or more collocated frames. Furthermore, if the second frame index of a particular candidate block corresponding to the second reference list List1 is equal to zero, the second reference frame of the particular candidate block selected from the second reference list List1 based on the second frame index may also be included in one or more collocated frames. However, if the frame index in the frame information of the particular candidate block is different from at least one predetermined value, the reference frame of the particular candidate block may be excluded from one or more collocated frames. Thus, the number of reference frames of the particular candidate block added to one or more collocated frames may be equal to 0, 1, or 2 when the particular candidate block is a bi-predictive block.

[0080] In some embodiments, the decoder module 124 may determine a motion block using a particular one of the motion candidates corresponding to a particular candidate block when the reference frame selected by the frame information of the particular motion candidate is included in one or more collocated frames. The decoder module 124 may select a motion block from a reference frame indicated by the particular motion candidate based on the motion vector of the particular motion candidate. The motion block of the particular motion candidate in the reference frame is indicated by the motion vector of the particular motion candidate from the block unit. In this way, the decoder module 124 may determine a motion block for one of the frame indexes when one of the frame indexes in the motion candidate is equal to one of the at least one predetermined value. The number of motion blocks may be equal to the number of frame indexes of the motion candidate equal to one of the at least one predetermined value. Each of the motion blocks can be considered as a collocated block. Thus, each of the collocated blocks may be selected from the reference frame of the corresponding one of the candidate blocks based on the corresponding one of the motion vectors and the corresponding one of the candidate positions.

[0081] In some embodiments, the decoder module 124 may directly use the motion vector of the motion candidate to determine the motion block in each of the one or more collocated frames. The one or more collocated frames may be directly selected from the first reference list List0 and the second reference list List1 based on the at least one predetermined value. The motion block selected by a particular one of the motion candidates is indicated in each of the one or more collocated frames by the motion vector of the particular motion candidate from the block unit. Thus, the particular motion candidate may be used to generate multiple motion blocks each included in a different collocated frame.

[0082] In some embodiments, since the number of motion blocks generated based on the motion vector in a particular motion candidate may be greater than one, the decoder module 124 may determine a motion cost value for each of the motion blocks generated in different collocated frames based on the same motion vector. The motion cost value may be determined by a template matching cost function. The decoder module 124 may determine a motion area for each of the motion blocks generated based on the same motion vector and compare the motion area to a template area of ​​a block unit. The decoder module 124 may then compare the motion cost values ​​of the motion blocks generated in different collocated frames based on the same motion vector with each other to select only one of the one or more collocated frames that includes one of the motion blocks of the motion vector of the particular motion candidate. In this manner, one of the motion blocks of the particular motion candidate may be set as a collocated block for further comparison with other collocated blocks of other motion candidates. Thus, the number of collocated blocks may be equal to the number of frame indexes that is equal to one of at least one predetermined value.

[0083] In some embodiments, motion blocks generated in different collocated frames based on the same motion vector may not be compared with each other. The decoder module 124 may generate all motion blocks in one or more collocated frames based on all motion candidates, and set all motion blocks as collocated blocks. Thus, the number of motion vectors for determining the collocated blocks may be equal to the number of indexes of frames equal to one of at least one predetermined value, so that the number of collocated blocks may be equal to an amount generated by multiplying the number of indexes of frames equal to one of at least one predetermined value by the number of one or more collocated frames.

[0084] Returning to FIG. 3, in block 360, the decoder module 124 determines a first cost value for each of the multiple collocated block units.

[0085] 1 and 2, the decoder module 124 may determine a plurality of collocated regions from one or more collocated frames, each of which is adjacent to one of the plurality of collocated blocks. Each of the collocated regions may include at least one of the plurality of adjacent regions of a corresponding one of the collocated blocks. A first adjacent region of a particular one of the collocated blocks may be an upper adjacent region located on the upper side of the collocated neighboring block, a second adjacent region of the particular collocated block may be a left adjacent region located on the left side of the particular collocated block, and a third adjacent region of the particular collocated block may be an upper left adjacent region located on the upper left of the particular collocated block. Since each of the one or more collocated frames is reconstructed before reconstruction of the block unit, the adjacent regions of the collocated blocks may be reconstructed before reconstruction of the block unit.

[0086] 6 is an illustration of an example embodiment of an image frame 60 having a block unit 600 and a template region 6000, and a collocated frame 65-66 having collocated blocks 650-660 and collocated regions 6500-6600, according to an embodiment of the present disclosure. The size of the collocated blocks 650 and 660 may be the same as the size of the block unit 600, and the size of the collocated regions 6500 and 6600 may be the same as the size of the template region 6000. Furthermore, the samples in the collocated blocks 650 and 660 and the collocated regions 6500 and 6600 may also be reconstructed samples that have been reconstructed before the block unit 600.

[0087] The decoder module 124 may determine a collocated cost value based on each of the template region and the multiple collocated regions by using a cost function. In FIG. 6, the decoder module 124 may directly compare the collocated region 6500 of the collocated block 650 and the collocated region 6600 of the collocated block 660 with the template region 6000 of the block unit 600 by the cost function. For example, the decoder module 124 may compare multiple collocated samples in the collocated region 6500 of the neighboring block 650 with the reconstructed samples in the template region 6000 of the block unit 600 to generate one of the collocated cost values.

[0088] The cost function may be a template matching cost function including, but not limited to, SAD, SAT, MAD, MSD, and SSIM. It should be noted that any cost function may be used without departing from this disclosure.

[0089] In order to distinguish the collocated cost value, the neighboring cost value, the averaged cost value, and the motion cost value from each other, the collocated cost value may be regarded as a first cost value, the neighboring cost value may be regarded as a plurality of second cost values, the averaged cost value may be regarded as a plurality of third cost values, and the motion cost value may be regarded as a plurality of fourth cost values. The second cost value and the third cost value may be used only for determining motion candidates in some embodiments of the method 300, so the second cost value and the third cost value may not be used in the method 300. The fourth cost value may be used for determining collocated blocks in some embodiments of the method 300, so the fourth cost value may not be used in the method 300. In some embodiments, the second cost value, the third cost value, and the fourth cost value may all be used in the method 300 for determining motion candidates and determining collocated blocks.

[0090] At block 370, the decoder module 124 determines at least one sub-block-based temporal motion vector prediction (SbTMVP) candidate based on the multiple motion candidates and the multiple first cost values.

[0091] The decoder module 124 may select at least one of the collocated blocks based on the first cost value calculated by the collocated region. When the number of the selected at least one collocated block is equal to four, the four collocated blocks having the four lowest first cost values ​​may be selected, and the decoder module 124 may determine the motion information of the four collocated blocks as four collocated candidates. When the number of the selected at least one collocated block is equal to one, the collocated block having the lowest collocated cost value may be selected, and the decoder module 124 may determine the motion information of the selected collocated block as a collocated candidate. Thus, when the number of the selected at least one collocated block is equal to Y, Y collocated blocks may be selected based on the Y lowest collocated cost values ​​of the collocated cost values ​​to determine Y collocated candidates. The number Y may be a positive integer. Each of the at least one collocated candidate may indicate motion information, list information, and frame information for a corresponding one of the at least one collocated blocks.

[0092] Each of the at least one SbTMVP candidates may be determined based on a corresponding one of the at least one collocated candidate. The decoder module 124 may determine a collocated distance between a reference frame of a particular one of the selected at least one collocated block and a particular one of the collocated frames including the particular selected collocated block. The reference frame of the particular selected collocated block may be indicated by the motion information (e.g., the collocated candidate) of the particular selected collocated block. Furthermore, the decoder module 124 may determine a reference distance between the reference frame of the block unit and the image frame. Then, the motion information of the particular one of the at least one SbTMVP candidates may be determined based on the corresponding one of the collocated distances, the corresponding one of the reference distances, and the motion information of the corresponding one of the collocated candidates. In some embodiments, the reference frame of the block unit may be a corresponding one of the one or more collocated blocks for determining the corresponding one of the at least one SbTMVP candidate.

[0093] Returning to FIG. 3, in block 380, the decoder module 124 reconstructs block units based on a candidate list that includes at least one SbTMVP candidate.

[0094] 1 and 2, the decoder module 124 may select a prediction candidate from the candidate list based on a prediction index. The prediction index may indicate a prediction candidate for the block unit from the candidate list. Thus, the decoder module 124 may generate a prediction block for the block unit based on the prediction candidate selected from the candidate list. In some embodiments, the candidate list may be a merge list including multiple merge candidates, and at least one SbTMVP candidate may be included in the merge candidate. In some embodiments, the candidate list may be a subblock merge list including multiple subblock merge candidates, and at least one SbTMVP candidate may be included in the subblock merge candidate. In some embodiments, the prediction index may be an index SbTMVP_idx for determining a selected one of the SbTMVP candidates when the candidate list includes only SbTMVP candidates.

[0095] The decoder module 124 may further add a plurality of residual components to the prediction block to reconstruct the block unit, which may be determined from the bitstream. The decoder module 124 may reconstruct all other block units in the image frame to reconstruct the image frame and the video data.

[0096] Another method M1, similar to method 300, may be performed using the configurations shown in FIGS. 1 and 2, the various elements of which are referenced with respect to method 300.

[0097] In method M1, the decoder module 124 may decode the video data based on the method 300. Further, in block 370, after selecting at least one of the collocated blocks, the decoder module 124 further determines a plurality of shift blocks each adjacent to one of the selected at least one collocated blocks. In some embodiments, each of the shift blocks may be selected based on a plurality of neighboring positions including a plurality of neighboring positions and a plurality of non-adjacent positions in FIG. 4. In some embodiments, each of the shift blocks may be selected based on a plurality of neighboring positions including four directional positions (e.g., an upper position, a lower position, a left position, and a right position). In some embodiments, each of the shift blocks may be selected based on a plurality of neighboring positions including eight directional positions (e.g., an upper position, a lower position, a left position, a right position, an upper left position, an upper right position, a lower left position, and a lower right position).

[0098] 7 is an illustration of an example embodiment of an image frame 70 having a block unit 700 and a template region 7000, and a collocated frame 77 having a collocated block 770, shift blocks 771-774, and collocated regions 7700-7740, according to an embodiment of the present disclosure. The size of the collocated block 770 and the shift blocks 771-774 may be the same as the size of the block unit 700, and the size of the collocated regions 7700-7740 may be the same as the size of the template region 7000. Furthermore, the samples in the collocated block 770, the shift blocks 771-774, and the collocated regions 7700-7740 may also be reconstructed samples that are reconstructed before the block unit 700.

[0099] The decoder module 124 may determine a shift cost value based on the template region and each of the plurality of collocated regions by using a cost function. In FIG. 7, the decoder module 124 may directly compare the template region 7000 of the block unit 700 with each of the collocated regions 7700-7740 by the cost function. For example, the decoder module 124 may compare the plurality of collocated samples in the collocated region 7740 of the shift block 740 with the reconstructed samples in the template region 7000 of the block unit 700 to generate one of the shift cost values. If one of the shift cost values ​​determined based on the collocated region of the shift block is smaller than the shift cost value determined based on the collocated region of the collocated block, the corresponding one of the shift blocks may replace the collocated block to determine a collocated candidate. In this manner, the collocated candidate may indicate the motion vector, list information, and frame information of the corresponding one of the shift blocks.

[0100] 8 illustrates a flowchart of a method 800 for decoding video data by an electronic device according to an embodiment of the present disclosure. The method 800 is merely exemplary, as there are various ways to perform the decoding of video data.

[0101] Method 800 may be performed using the configurations illustrated in Figures 1 and 2, where various elements are referenced with respect to method 300. Each block illustrated in Figure 8 may represent one or more processes, methods, or subroutines that are performed.

[0102] The order of the blocks in Figure 8 is exemplary and may be changed: additional blocks may be added or fewer blocks may be utilized without departing from this disclosure.

[0103] At block 810, the decoder module 124 receives the video data. The video data received by the decoder module 124 may be a bitstream.

[0104] 1 and 2, the second electronic device 120 may receive a bitstream from an encoder, such as the first electronic device 110, or another video provider, via a second interface 126. The second interface 126 may provide the bitstream to a decoder module 124.

[0105] The entropy decoding unit 2241 may decode the bitstream to determine prediction indications and partition indications for the image frames. The decoder module 124 may then further reconstruct the image frames based on the prediction indications and partition indications. The prediction indications and partition indications may include flags and indexes.

[0106] At block 820, the decoder module 124 determines block units from the image frame according to the video data.

[0107] 1 and 2, the decoder module 124 may determine an image frame based on a bitstream, and divide the image frame to determine block units according to a partition instruction in the bitstream. For example, the decoder module 124 may divide the image frame to generate multiple CTUs, and further divide one of the CTUs to determine block units according to a partition instruction based on any video coding standard.

[0108] At block 830, the decoder module 124 determines a number of candidate locations selected from a number of adjacent locations adjacent to the block unit and a number of non-adjacent locations not adjacent to the block unit.

[0109] The candidate locations may be a plurality of proximate locations adjacent to the block unit. The proximate locations may be selected from at least one of a plurality of proximate locations adjacent to the block unit or a plurality of non-proximate locations not adjacent to the block unit. Thus, in some embodiments, the candidate locations may include at least one of a proximate location or a non-proximate location. In some embodiments, the candidate locations may include at least one of a proximate location and at least one of a non-proximate location. In some embodiments, the candidate locations may include at least one of a non-proximate location and exclude a proximate location. In some embodiments, the candidate locations may exclude a non-proximate location and include at least one of a proximate location.

[0110] 1, 2 and 4, the decoder module 124 may determine a number of adjacent positions 401-405 adjacent to the current block 400 and a number of non-adjacent positions 406-423 not adjacent to the block unit 400. Each of the adjacent positions 401-405 may be adjacent to one of the four corners of the block unit 400. Furthermore, the distance between the block unit 400 and one of the non-adjacent positions 406-423 may be based on one of the block height H, block width W, or diagonal length D of the block unit 400. For example, the horizontal and vertical distances between the top left position of the block unit 400 and the non-adjacent position 408 may be equal to W+1 and H+1 of the block unit 400, respectively, and the distance between the block unit 400 and the non-adjacent position 411 may be equal to a distance value generated by adding one to two times the block height H of the block unit 400.

[0111] At block 840, the decoder module 124 determines multiple motion candidates from the multiple candidate positions.

[0112] 1 and 2, the decoder module 124 may determine a plurality of neighboring blocks adjacent to the block unit based on the candidate positions. Each of the neighboring blocks may cover at least one of the candidate positions. Each of the candidate positions may be located at an upper left position of a corresponding one of the neighboring blocks. Furthermore, the size of the neighboring blocks may be the same as the size of the block unit (i.e., the block height H, the block width W, or the diagonal length D).

[0113] The neighborhood block determined based on one of the adjacent locations may cover one or more of the adjacent locations, since some of the adjacent locations are adjacent to each other. The neighborhood block determined based on one of the non-adjacent locations may cover one of the non-adjacent locations. Furthermore, the neighborhood block determined based on one of the non-adjacent locations may cover only one of the non-adjacent locations, since the distance between the non-adjacent locations may be equal to or larger than the size of a block unit.

[0114] The decoder module 124 may determine a template region adjacent to the block unit from the image frame. The template region may include at least one of a plurality of adjacent regions. The first adjacent region may be an adjacent region located on the top side of the block unit, the second adjacent region may be an adjacent region located on the left side of the block unit, and the third adjacent region may be an adjacent region located on the top left side of the block unit. The adjacent regions may be reconstructed before reconstructing the block unit. The height of the first adjacent region may be equal to the number R of reconstructed samples of the first adjacent region along the vertical direction, and the width of the first adjacent region may be equal to the width of the block unit. The height of the second adjacent region may be equal to the height of the block unit, and the width of the second adjacent region may be equal to the number S of reconstructed samples of the second adjacent region along the horizontal direction. Furthermore, the height of the third adjacent region may be equal to the number R of reconstructed samples of the first adjacent region along the vertical direction, and the width of the third adjacent region may be equal to the number S of reconstructed samples of the second adjacent region along the horizontal direction. In one embodiment, the numbers R and S may be positive integers. Furthermore, the numbers R and S may be equal to or different from each other. Furthermore, the numbers R and S may be greater than or equal to 1. In some embodiments, the numbers R and S may be equal to 1.

[0115] The decoder module 124 may determine template regions adjacent to the block unit. The decoder module 124 may use the adjacent regions as template regions for determining the motion candidate. Furthermore, the decoder module 124 may use two of the adjacent regions as template regions for determining the motion candidate. For example, the decoder module 124 may use only the first and second adjacent regions as template regions for determining the motion candidate. Furthermore, the decoder module 124 may use only one of the adjacent regions as template regions for determining the motion candidate.

[0116] The decoder module 124 may determine a plurality of neighboring regions from the image frame. Each of the neighboring regions may be adjacent to a corresponding one of the neighboring blocks. Each of the neighboring regions may include at least one of the plurality of neighboring regions of the corresponding one of the neighboring blocks. The first neighboring region of a particular one of the neighboring blocks may be a neighboring region located on the upper side of the particular neighboring block, the second neighboring region of the particular neighboring block may be a neighboring region located on the left side of the particular neighboring block, and the third neighboring region of the particular neighboring block may be a neighboring region located on the upper left side of the particular neighboring block. The neighboring regions of the neighboring blocks may be reconstructed before reconstructing the block unit.

[0117] 5A, the size of the neighborhood block 530 may be the same as the size of the block unit 500. Since the neighborhood block 530 is located above and to the left of the block unit 500 and does not cover the block unit 500, the samples in the neighborhood block 530 may be reconstructed samples reconstructed before the block unit 500. Furthermore, the samples in the neighborhood region 5300 may also be reconstructed samples reconstructed before the block unit 500.

[0118] Referring to FIG. 5B, the size of the neighborhood block 540 may be the same as the size of the block unit 500. However, since the neighborhood position 502 is located adjacent to the upper right corner of the block unit 500, some of the samples in the neighborhood block 540 may be reconstructed samples reconstructed before the block unit 500, and other samples in the neighborhood block 540 may be non-reconstructed samples. Therefore, the decoder module 124 may determine a neighborhood region 5400 that is smaller than the template region 5000 of FIG. 5A to exclude the area covered by the block unit 500. Thus, since the neighborhood region 5400 is located above the block unit 500 and does not cover the block unit 500, the samples in the neighborhood region 5400 may also be reconstructed samples reconstructed before the block unit 500.

[0119] The decoder module 124 may determine a neighborhood cost value based on the template region and each of the multiple neighborhood regions using a cost function to select a motion candidate for the block unit. In FIG. 5A, the decoder module 124 may directly compare the neighborhood region 5300 of the neighboring block 530 with the template region 5000 of the block unit 500 by the cost function. For example, the decoder module 124 may compare multiple adjacent samples in the neighborhood region 5300 of the neighboring block 530 with the reconstructed samples in the template region 5000 of the block unit 500 to generate one of the neighborhood cost values. However, since the neighborhood region 5400 may be smaller than the template region 5000, the decoder module 124 may not directly compare the neighborhood region 5400 of the neighboring block 540 with the template region 5000 of the block unit 500 by the cost function. Referring to FIG. 5C, the secondary region 5001 may be determined based on the size and shape of the neighborhood region 5400. The size and shape of the secondary region 5001 may be identical to the size and shape of the neighborhood region 5400. Therefore, the decoder module 124 may directly compare the neighborhood region 5400 of the neighboring block 540 with the template region 5001 of the block unit 500 by a cost function.

[0120] The cost function may be a template matching cost function including, but not limited to, sum of absolute differences (SAD), sum of absolute transformed differences (SATD), mean absolute difference (MAD), mean squared difference (MSD), and structural similarity measure (SSIM). It should be noted that any cost function may be used without departing from this disclosure.

[0121] The decoder module 124 may select multiple candidate blocks based on the neighborhood cost values ​​calculated by the neighborhood regions. However, since the sizes of the neighborhood regions may be the same or different from each other, it may be unfair to directly compare the neighborhood cost values ​​with each other. The decoder module 124 may determine an average cost value for each of the neighborhood blocks based on the size of the neighborhood region. For example, the average cost value of the neighborhood block 530 may be calculated by dividing the neighborhood cost value of the neighborhood region 5300 by 8, and the average cost value of the neighborhood block 540 may be calculated by dividing the neighborhood cost value of the neighborhood region 5400 by 5. In this manner, the decoder module 124 may compare the average cost values ​​with each other to select some of the neighborhood positions for determining the candidate blocks.

[0122] If the number of selected neighborhood positions is equal to 3, three neighborhood positions corresponding to three neighborhood blocks with the three lowest averaged cost values ​​may be selected, and the motion candidates may include motion information of three candidate blocks covering the three neighborhood positions. If the number of selected neighborhood positions is equal to 1, one neighborhood position corresponding to one neighborhood block with the lowest averaged cost value may be selected, and the motion candidates may include motion information of one candidate block covering the one neighborhood position. Thus, if the number of selected neighborhood positions is equal to Y, Y neighborhood positions may be selected based on the Y lowest averaged cost values, and Y candidate blocks may be determined. The number Y may be a positive integer.

[0123] The candidate block may be a block reconstructed before reconstructing the block unit. When one of the candidate blocks does not have motion information, the motion candidate may not be determined based on the corresponding one selected neighboring position. For example, when one of the candidate blocks is an intra-prediction block, one of the candidate blocks does not have motion information to be added to the motion candidate. In addition, the motion candidate may further include a history-based candidate determined from a first-in-first-out (FIFO) table that stores multiple previous motion information of multiple previous blocks reconstructed before reconstructing the block unit.

[0124] Returning to FIG. 8, at block 850, the decoder module 124 determines, based on the motion candidates, a number of collocated blocks, each of which is included in one of the one or more collocated frames.

[0125] Each of the multiple motion candidates may indicate a motion vector, list information, and frame information of a candidate block that covers one of the multiple candidate positions. Furthermore, the motion vector, list information, and frame information of a particular one of the candidate blocks may be used to predict the particular candidate block in order to reconstruct the particular candidate block.

[0126] The list information of the particular candidate block may include at least one of an inter prediction indication or a multiple list prediction flag. The inter prediction indication may indicate which of a first uni-prediction with a first reference list List0, a second uni-prediction with a second reference list List1, and a bi-prediction including a first reference list List0 and a second reference list List1 is used for the particular candidate block. Furthermore, the list prediction flag may include a first list prediction flag used to determine whether a first reference list List0 is used for the particular candidate block, and a second list prediction flag used to determine whether a second reference list List1 is used for the particular candidate block. In some embodiments, the list information of the particular candidate block may include both an inter prediction indication and a multiple list prediction flag for determining a reference list of the particular candidate block. Furthermore, the list information of the particular candidate block may include only an inter prediction indication for determining a reference list of the particular candidate block. Furthermore, the list information of the particular candidate block may include only a multiple list prediction flag for determining a reference list of the particular candidate block.

[0127] The frame information may include at least one frame index. When a particular candidate block is a bi-predictive block, the frame information of the particular candidate block may include two frame indexes to indicate two reference frames respectively included in the corresponding one of the reference lists. Furthermore, when a particular candidate block is a uni-predictive block, the frame information of the particular candidate block may include only one frame index to indicate one reference frame included in the corresponding one of the reference lists. In another embodiment, when a particular candidate block is a uni-predictive block, the frame information of the particular candidate block may also include two frame indexes. One of the two frame indexes may be used to indicate one reference frame included in the corresponding one of the reference lists, and the other of the two frame indexes may be equal to a predetermined frame value to indicate that the other one of the reference lists is not used for the particular candidate block.

[0128] 1 and 2, the decoder module 124 may determine one or more collocated frames based on at least one predetermined value. If one of the frame indexes in the frame information of the particular candidate block is equal to one of the at least one predetermined value, the reference frame of the particular candidate block selected from the corresponding reference list indicated by the list information of the particular candidate block may be included in one or more collocated frames. For example, the at least one predetermined value may include zero. In some embodiments, the particular candidate block may be a single predicted block predicted based on the second reference list List1. If the second frame index of the particular candidate block corresponding to the second reference list List1 is equal to zero, the reference frame of the particular candidate block selected from the second reference list List1 based on the second frame index may be included in one or more collocated frames. However, the reference frame of the particular candidate block selected from the second reference list List1 may be excluded from one or more collocated frames if the second frame index in the frame information of the particular candidate block is different from the at least one predetermined value. In this manner, the reference frame of a particular candidate block may be added to or excluded from one or more collocated frames based on the frame information.

[0129] In some embodiments, the particular candidate block may be a bi-predictive block predicted based on the first reference list List0 and the second reference list List1. Furthermore, some of the multiple reference candidates in the first reference list List0 may be identical to some of the multiple reference candidates in the second reference list List1. Thus, a first reference frame of the particular candidate block selected from the first reference list List0 based on the first frame index may be identical to a second reference frame of the particular candidate block selected from the second reference list List1 based on the second frame index. Thus, the number of reference frames of the particular candidate block determined from the two reference lists may be equal to 1 or 2 when the particular candidate block is a bi-predictive block.

[0130] If the first frame index of a particular candidate block corresponding to the first reference list List0 is equal to zero, the first reference frame of the particular candidate block selected from the first reference list List0 based on the first frame index may be included in one or more collocated frames. Furthermore, if the second frame index of a particular candidate block corresponding to the second reference list List1 is equal to zero, the second reference frame of the particular candidate block selected from the second reference list List1 based on the second frame index may also be included in one or more collocated frames. However, if the frame index in the frame information of the particular candidate block is different from the at least one predetermined value, the reference frame of the particular candidate block may be excluded from one or more collocated frames. Thus, the number of reference frames of the particular candidate block added to one or more collocated frames may be equal to 0, 1, or 2 when the particular candidate block is a bi-predictive block.

[0131] In some embodiments, the decoder module 124 may determine a motion block using a particular one of the motion candidates corresponding to a particular candidate block when the reference frame selected by the frame information of the particular motion candidate is included in one or more collocated frames. The decoder module 124 may select a motion block from a reference frame indicated by the particular motion candidate based on the motion vector of the particular motion candidate. The motion block of the particular motion candidate in the reference frame is indicated by the motion vector of the particular motion candidate from the block unit. In this way, the decoder module 124 may determine a motion block for one of the frame indexes when one of the frame indexes in the motion candidate is equal to one of the at least one predetermined value. The number of motion blocks may be equal to the number of frame indexes of the motion candidate equal to one of the at least one predetermined value. Each of the motion blocks can be considered as a collocated block. Thus, each of the collocated blocks may be selected from the reference frame of the corresponding one of the candidate blocks based on the corresponding one of the motion vectors and the corresponding one of the candidate positions.

[0132] In some embodiments, the decoder module 124 may directly use the motion vector of the motion candidate to determine the motion block in each of the one or more collocated frames. The one or more collocated frames may be directly selected from the first reference list List0 and the second reference list List1 based on the at least one predetermined value. The motion block selected by a particular one of the motion candidates is indicated in each of the one or more collocated frames by the motion vector of the particular motion candidate from the block unit. Thus, the particular motion candidate may be used to generate multiple motion blocks each included in a different collocated frame.

[0133] In some embodiments, since the number of motion blocks generated based on the motion vector in a particular motion candidate may be greater than one, the decoder module 124 may determine a motion cost value for each of the motion blocks generated in different collocated frames based on the same motion vector. The motion cost value may be determined by a template matching cost function. The decoder module 124 may determine a motion area for each of the motion blocks generated based on the same motion vector and compare the motion area with a template area of ​​a block unit. The decoder module 124 may then compare the motion cost values ​​of the motion blocks generated in different collocated frames based on the same motion vector with each other to select only one of the one or more collocated frames that includes one of the motion blocks of the motion vector of the particular motion candidate. In this way, one of the motion blocks of the particular motion candidate may be set as a collocated block for further comparison with other collocated blocks of other motion candidates. Thus, the number of collocated blocks may be equal to the number of frame indexes that is equal to one of the at least one predetermined value.

[0134] In some embodiments, motion blocks generated in different collocated frames based on the same motion vector may not be compared with each other. The decoder module 124 may generate all motion blocks in one or more collocated frames based on all motion candidates, and set all motion blocks as collocated blocks. Thus, since the number of motion vectors for determining collocated blocks is equal to the number of indexes of frames equal to one of the at least one predetermined value, the number of collocated blocks may be equal to an amount generated by multiplying one of the at least one predetermined value by the number of one or more collocated frames.

[0135] Returning to FIG. 8, in block 860, the decoder module 124 derives multiple sub-block-based temporal motion vector prediction (SbTMVP) candidates based on the motion information of the multiple collocated blocks.

[0136] 1 and 2, the decoder module 124 may determine whether the collocated block has motion information. If the collocated block is predicted or reconstructed by intra prediction, the collocated block may only include an intra prediction mode that does not include motion information. If the collocated block is predicted or reconstructed by inter prediction, the decoder module 124 may receive the motion information of the collocated block.

[0137] The motion information of each of the collocated blocks may include vector information, list information, and frame information of a corresponding one of the collocated blocks. Furthermore, the vector information, list information, and frame information of a particular one of the collocated blocks may be used to predict a particular collocated block in order to reconstruct the particular collocated block.

[0138] The decoder module 124 may use the motion information of the collocated blocks to determine SbTMVP candidates, each of which corresponds to one of the collocated blocks. In some embodiments, the decoder module 124 may determine an initial SbTMVP list including all of the SbTMVP candidates of the block unit. When the block unit is predicted based on the SbTMVP candidates, the block unit may be divided into a plurality of subblock units. Thus, each of the collocated blocks may also be divided into a plurality of collocated subblocks. The decoder module 124 may use the motion information of the collocated blocks to determine a plurality of first subblock vectors Vs1 for the collocated subblocks as the collocated frame is reconstructed before reconstructing the image frame. The decoder module 124 may then derive a plurality of second subblock vectors Vs2 for the subblock unit based on the plurality of first subblock vectors Vs1. For example, the decoder module 124 may use the motion information of a particular collocated block to determine M first subblock vectors Vs1 for the M collocated subblocks in the particular collocated block. The decoder module 124 may then derive M second subblock vectors Vs2 for the M subblock units in the block unit based on the M first subblock vectors Vs1. Each of the M second subblock vectors Vs2 may be derived based on a corresponding one of the M first subblock vectors Vs1, a reference distance between the image frame and the reference frame of the block unit, and a collocated distance between the collocated frame and the collocated reference frame of the collocated block.

[0139] Furthermore, when a particular one of the collocated subblocks is included in a uni-predictive block, the decoder module 124 may derive one second subblock vector for a particular one of the subblock units corresponding to the particular collocated subblock. Furthermore, when a particular collocated subblock is included in a bi-predictive block, the decoder module 124 may derive two second subblock vectors for a particular subblock unit corresponding to the particular collocated subblock. In some embodiments, some of the collocated subblocks in a particular collocated block may be uni-predictive blocks, and others of the collocated subblocks in a particular collocated block may be bi-predictive blocks. Thus, when the decoder module 124 determines the M collocated subblocks in a particular collocated block, the decoder module 124 may use the motion information of the M collocated subblocks to determine the (M+X) first subblock vectors Vs1 of the M collocated subblocks in the particular collocated block. The decoder module 124 may then derive (M+X) second sub-block vectors Vs2 for the M sub-block units in the block unit based on the (M+X) first sub-block vectors Vs1. The number X may be equal to the number of bi-predictive collocated sub-blocks in a particular collocated block. The number X may be in the range of 0 to M. Thus, the number of second sub-block vectors Vs2 may be in the range of M to 2M.

[0140] Since the M first subblock vectors Vs1 of the collocated subblocks received from the differential encoding unit may differ from each other, the M second subblock vectors Vs2 of the subblock unit may differ from each other. Thus, when the decoder module 124 uses the M second subblock vectors Vs2 determined from a particular one of the collocated blocks to determine one of the SbTMVP candidates, the decoder module 124 may determine multiple reference subblocks that are not adjacent to each other, since the M second subblock vectors Vs2 corresponding to a particular collocated block may differ from each other. In some embodiments, each of the SbTMVP candidates may indicate the M first subblock vectors Vs1 for a corresponding one of the collocated blocks. In some embodiments, each of the SbTMVP candidates may indicate the M second subblock vectors Vs2 corresponding to one of the collocated blocks for the block unit. In some embodiments, each of the SbTMVP candidates may indicate a reference sub-block corresponding to one of the collocated blocks for the block unit.

[0141] Returning to FIG. 8, at block 870, the decoder module 124 determines a first cost value based on each of the multiple SbTMVP candidates.

[0142] 1 and 2, the decoder module 124 may determine a template region adjacent to the block unit from the image frame. The template region may include at least one of a plurality of adjacent regions. The first adjacent region may be an adjacent region located on the upper side of the block unit, the second adjacent region may be an adjacent region located on the left side of the block unit, and the third adjacent region may be an adjacent region located on the upper left side of the block unit. The adjacent regions may be reconstructed before reconstructing the block unit.

[0143] The decoder module 124 may further determine a reference region including a plurality of reference sub-regions generated for a reference sub-block corresponding to one of the collocated blocks. Since the reference sub-blocks corresponding to a particular one of the collocated blocks may not be adjacent to each other, the reference sub-region corresponding to the particular collocated block may be determined based on a portion of the M second sub-block vectors Vs2. The portion of the M second sub-block vectors Vs2 may belong to a sub-block unit located at the upper boundary of the block unit, or may belong to a sub-block unit located at the left boundary of the block unit.

[0144] 9A and 9B are diagrams illustrating an example implementation of a template region and a reference region according to an embodiment of the present disclosure. FIG. 9A is a diagram illustrating an example implementation of a block unit 900 and a template region 900 according to an embodiment of the present disclosure. A template region 9000 adjacent to a block unit 900 is determined in an image frame. The template region 9000 may be reconstructed before reconstructing the block unit 900.

[0145] 9B is an explanatory diagram of an exemplary embodiment of a reference sub-block 901-916 and a reference region according to an embodiment of the present disclosure. The reference region may include a plurality of reference sub-regions 9011, 9051, 9091, 9131, 9132, 9142, 9152, and 9162. The reference sub-regions 9011, 9051, 9091, and 9131 may be located on the left side of the reference sub-blocks 901, 905, 909, and 913, respectively. The reference sub-regions 9132, 9142, 9152, and 9162 may be located on the upper side of the reference sub-blocks 913-916, respectively. The reference sub-blocks 901, 905, 909, and 913 may be determined based on the sub-block unit located at the left boundary of the block unit 900 and the second sub-block vector Vs2 of the sub-block unit located at the left boundary. The reference sub-blocks 913 to 916 may be determined based on the sub-block unit located at the upper boundary of the block unit 900 and the second sub-block vector Vs2 of the sub-block unit located at the upper boundary.

[0146] The decoder module 124 may determine a reference cost value based on the template region and each of the multiple reference regions using a cost function. The decoder module 124 may directly compare the template region 9000 of the block unit 900 with the reference regions that respectively correspond to one of the collocated blocks by the cost function. For example, the decoder module 124 may compare a reconstructed sample in a particular one of the reference regions that corresponds to a particular one of the collocated blocks with a reconstructed sample in the template region 9000 of the block unit 900 to generate one of the reference cost values.

[0147] The cost function may be a template matching cost function including, but not limited to, SAD, SAT, MAD, MSD, and SSIM. It should be noted that any cost function may be used without departing from this disclosure.

[0148] In order to distinguish the reference cost value, the neighborhood cost value, the averaged cost value, and the motion cost value from each other, the reference cost value may be a first cost value, the neighborhood cost value may be a plurality of second cost values, the averaged cost value may be a plurality of third cost values, and the motion cost value may be a plurality of fourth cost values. Since the second cost value and the third cost value may be used only for determining motion candidates in some embodiments of the method 800, it may be unnecessary to use the second cost value and the third cost value in the method 800. Since the fourth cost value may be used for determining collocated blocks in some embodiments of the method 800, it may be unnecessary to use the fourth cost value in the method 800. In some embodiments, the second cost value, the third cost value, and the fourth cost value may all be used in the method 800 for determining motion candidates and determining collocated blocks.

[0149] At block 880, the decoder module 124 determines an arrangement of the multiple SbTMVP candidates based on the first cost value and generates an adjusted SbTMVP list based on the arrangement.

[0150] 1 and 2, the decoder module 124 may determine an arrangement of the SbTMVP candidates based on the first cost value and sort the SbTMVP candidates based on the arrangement. In some embodiments, the SbTMVP candidates may be sorted in ascending or descending order of the first cost value.

[0151] Before the placement is determined based on the first cost value, the SbTMVP candidates may be ordered based on any rule, for example, based on the order of adjacent and non-adjacent positions.

[0152] The SbTMVP candidates may be sorted in ascending order of reference cost value. Thus, if the reference cost value of a particular one of the SbTMVP candidates is smaller than the reference cost values ​​of the other SbTMVP candidates, the particular SbTMVP candidate may be advanced to become the first SbTMVP candidate based on the arrangement. In other words, if the reference cost value of the particular SbTMVP candidate is the minimum of the reference cost values ​​of the SbTMVP candidates, the particular SbTMVP candidate may be advanced to the first SbTMVP candidate. Furthermore, if the reference cost value of the particular SbTMVP candidate is the maximum of the reference cost values ​​of the SbTMVP candidates, the particular SbTMVP candidate may be moved to the last one of the SbTMVP candidates. For example, if the number of SbTMVP candidates is equal to six, the SbTMVP candidates may include six SbTMVP candidates SC1, SC2, SC3, SC4, SC5, SC6 having six reference cost values ​​(CV1, CV2, CV3, CV4, CV5, CV6). In addition, if the value order of the six reference cost values ​​is CV4>CV2>CV5>CV1>CV6>CV3, the arrangement of the SbTMVP candidates may be changed from the original order of the six SbTMVP candidates SC1, SC2, SC3, SC4, SC5, and SC6 to a new order of the six SbTMVP candidates SC3, SC6, SC1, SC5, SC2, and SC4.

[0153] The arrangement may further be determined by adjusting the new order of the SbTMVP candidates based on the diversity reference. The decoder module 124 may determine a difference value between two of the reference cost values ​​selected from the two adjacent ones of the order of the SbTMVP candidates in the new order. If the difference value is equal to or less than the diversity threshold, the last one of the two adjacent ones of the SbTMVP candidates may be moved backward. For example, the value order of the six reference cost values ​​is CV4>CV2>CV5>CV1>CV6>CV3, and the difference between the two reference cost values ​​CV1 and CV6 is equal to or less than the diversity threshold. Therefore, the SbTMVP candidate SC1 may be moved backward. Thus, the arrangement of the SbTMVP candidates may further be changed from the new order of the six SbTMVP candidates SC3, SC6, SC1, SC5, SC2, SC4 to the final order of the six SbTMVP candidates SC3, SC6, SC5, SC1, SC2, SC4. In some embodiments, the arrangement may be identical to the new order of the SbTMVP candidates, since the arrangement may not be determined by adjusting the new order of the SbTMVP candidates based on a diversity criterion.

[0154] The decoder module 124 may select K SbTMVP candidates with the smallest reference cost values ​​from the SbTMVP candidates and add the selected SbTMVP candidates to the adjusted SbTMVP list. The number K, which is a positive integer, may be equal to the number of SbTMVP candidates in the adjusted SbTMVP list and may be less than the total amount of SbTMVP candidates. In other words, when the decoder module 124 sorts the SbTMVP candidates in ascending order of the reference cost values ​​to generate an arrangement, it may select the first to K SbTMVP candidates arranged based on the arrangement. In some embodiments, the adjusted SbTMVP list may differ from the initial SbTMVP list if the order of the initial cost values ​​is different from the original order of the SbTMVP candidates in the initial SbTMVP list. In some embodiments, the adjusted SbTMVP list may differ from the initial SbTMVP list if the number of the K selected SbTMVP candidates is different from the number of SbTMVP candidates in the initial SbTMVP list. In some embodiments, the adjusted SbTMVP list may be identical to the initial SbTMVP list if the order of the initial cost values ​​is the same as the original order of the SbTMVP candidates in the initial SbTMVP list and the number of K selected SbTMVP candidates is equal to the number of SbTMVP candidates in the initial SbTMVP list.

[0155] Each SbTMVP candidate in the adjusted SbTMVP list may have an SbTMVP index. Thus, since the number of SbTMVP candidates in the adjusted SbTMVP list is equal to K, the SbTMVP index of the adjusted SbTMVP list may be in the index range from 0 to K-1.

[0156] The SbTMVP candidates may be divided into multiple candidate groups based on multiple division criteria. For example, the SbTMVP candidates may be divided into multiple candidate groups based on the neighborhood position type, the block size of the block unit, or at least one predetermined number. When dividing the SbTMVP candidates based on the neighborhood position type, the SbTMVP candidates generated based on the non-adjacent positions may be included in one of the candidate groups, the SbTMVP candidates generated based on the adjacent positions may be included in another one of the candidate groups, and the SbTMVP candidates with a sub-block motion vector of zero may be included in the other one of the candidate groups. For example, the number of SbTMVP candidates may be equal to 10. There may be two SbTMVP candidates generated based on the adjacent positions, seven SbTMVP candidates generated based on the non-adjacent positions, and one SbTMVP candidate with a sub-block motion vector of zero. Thus, the SbTMVP candidates may be divided into three candidate groups, including a first candidate group with two SbTMVP candidates, a second candidate group with seven SbTMVP candidates, and a third candidate group with one SbTMVP candidate.

[0157] Further, if the SbTMVP candidates are divided based on the block size of the block unit, the decoder module 124 may determine a group size of the candidate group based on the block size of the block unit. For example, if the block size is 8×8, the group size may be equal to 3, and if the block size is 16×16, the group size may be equal to 5. Thus, the decoder module 124 may determine the group size based on the block size from the lookup table. Further, the group sizes of the candidate groups may be equal to each other. In some embodiments, the block size may be the width of the block unit. In some embodiments, the block size may be the height of the block unit. In some embodiments, the block size may be a size value generated by multiplying the width of the block unit by the height of the block unit. In some embodiments, the block size may be a size value generated by dividing the width of the block unit by the height of the block unit. In some embodiments, the block size may be a size value generated by dividing the height of the block unit by the width of the block unit. In some embodiments, the block size may be calculated based on the width of the block unit and / or the height of the block unit.

[0158] If the SbTMVP candidates are divided based on the at least one predetermined number, decoder module 124 may determine a group size of the candidate groups based on the at least one predetermined number. If the group sizes of the candidate groups are equal to each other, the at least one predetermined number may be equal to 1. If some of the group sizes of the candidate groups are different from each other, the at least one predetermined number may be greater than 1. Thus, decoder module 124 may divide the SbTMVP candidates into multiple candidate groups, each having the same or different group size. Furthermore, if the at least one predetermined number is equal to 1, the group size of the last candidate group may be different from the group size of the other candidate groups, since the number of SbTMVP candidates may not be divisible by the predetermined number. In some embodiments, the number of SbTMVP candidates may be equal to 5. The predetermined number for dividing the SbTMVP candidates may be equal to 3. The SbTMVP candidates in the first candidate group may be candidates SbTMVP0, SbTMVP1, and SbTMVP2 having corresponding indices 0, 1, and 2, respectively, and the SbTMVP candidates in the second candidate group may be candidates SbTMVP3 and SbTMVP4 having corresponding indices 3 and 4, respectively.

[0159] In some embodiments, the group sizes of the candidate groups may be different from each other. Thus, when the number of candidate groups is equal to x, the group sizes may be equal to N1, N2, N3, ..., Nx. For example, the SbTMVP candidates may be divided into two candidate groups. The first candidate group may include the first N1 SbTMVP candidates with indices from 0 to (N1-1), and the second candidate group may include the N2 SbTMVP candidates with indices from N1 to (N1+N2-1). In some embodiments, the number of SbTMVP candidates may be equal to 15. The predetermined number for dividing the SbTMVP candidates may be equal to 4, 5, and 6. Thus, the SbTMVP candidates may be divided into three candidate groups, each with a different group size. The first subgroup may include four SbTMVP candidates with indices from 0 to 3, and the second subgroup may include five SbTMVP candidates with indices from 4 to 8. The third subgroup may include six SbTMVP candidates with indices from 9 to 14.

[0160] The SbTMVP candidates may be collected based on a collection index used to derive how many SbTMVP candidates the decoder module 124 collects to select one of the SbTMVP candidates. For example, if the collection index indicates that the selected SbTMVP candidate is included in the pth candidate group, the number of SbTMVP candidates collected by the decoder module 124 may be greater than the sum of N1, N2, ..., Np. In other words, the decoder module 124 may collect only P SbTMVP candidates, and the amount of P SbTMVP candidates may be sufficient to distribute to the first pth candidate group. In some embodiments, the collection index may be a candidate group index indicating which of the candidate groups includes the selected SbTMVP candidate. In some embodiments, a prediction index that directly indicates the selected SbTMVP candidate may be used as the collection index to derive the number of collected SbTMVP candidates. For example, the number of SbTMVP candidates may be equal to 15, and the predetermined number for dividing the SbTMVP candidates may be equal to 4, 5, 6. The selected SbTMVP candidate may be included in the second candidate group if the analyzed prediction index is equal to 5. In this manner, the decoder module 124 may stop collecting SbTMVP candidates once there are 9 collected SbTMVP candidates.

[0161] In some embodiments, the SbTMVP candidates included in each of the candidate groups may be sorted in ascending order of the reference cost values ​​of the SbTMVP candidates included in the corresponding one of the candidate groups. Thus, if the reference cost value of a particular one of the SbTMVP candidates included in a particular one of the candidate groups is less than the reference cost values ​​of the other SbTMVP candidates included in the particular candidate group, the particular SbTMVP candidate may be advanced to become the first SbTMVP candidate in the particular candidate group based on the arrangement. In other words, if the reference cost value of the particular SbTMVP candidate in the particular candidate group is the minimum value of the reference cost values ​​of the particular candidate group, the particular SbTMVP candidate may be advanced to become the first SbTMVP candidate in the particular candidate group.

[0162] Returning to FIG. 8, in block 890, the decoder module 124 reconstructs the block units based on the adjusted SbTMVP list.

[0163] 1 and 2, the decoder module 124 may select one of the SbTMVP candidates ordered based on the arrangement from the adjusted SbTMVP list based on a prediction index. The prediction index may indicate a prediction candidate for the block unit from the SbTMVP candidates ordered based on the arrangement of the adjusted SbTMVP list. In some embodiments, the prediction index may be an index SbTMVP_idx for determining the selected one of the SbTMVP candidates.

[0164] The index value of the SbTMVP index in the adjusted SbTMVP list may be within the index range of 0 to K-1. Therefore, since the index value of the SbTMVP index may not be greater than K-1, the SbTMVP candidates arranged after the Kth one among the SbTMVP candidates arranged by the array may be excluded from the selection of the predicted candidates.

[0165] In some embodiments, the SbTMVP candidates ordered based on the placement in the adjusted SbTMVP list may be added to a candidate list. The decoder module 124 may select one of the multiple mode candidates including the SbTMVP candidates ordered based on the placement based on a prediction index. The prediction index may indicate a prediction candidate for a block unit. For example, the candidate list may be a subblock merge list, and the prediction index may be a subblock merge index. Additionally, the candidate list may be a merge list, and the prediction index may be a merge index.

[0166] The decoder module 124 may determine a selected one of the SbTMVP candidates based on an index in the bitstream and may determine a prediction candidate based on the selected SbTMVP candidate. The decoder module 124 may then generate a prediction block for the block unit based on the selected SbTMVP candidate.

[0167] The decoder module 124 may further add a plurality of residual components to the prediction block to reconstruct the block unit. The residual components may be determined from the bitstream. The decoder module 124 may reconstruct all other block units in the image frame to reconstruct the image frame and the video data.

[0168] Another method M2, generated by combining methods 300 and 800, may be performed using the configurations shown in Figures 1 and 2, the various elements of which are referenced with respect to methods 300 and 800.

[0169] In method M2, decoder module 124 may decode video data based on blocks 310-360 of method 300. Decoder module 124 may then select a portion of collocated blocks based on the collocated cost values ​​of the collocated blocks. For example, when the number of the portion of collocated blocks is equal to Y, Y collocated blocks may be selected based on the Y lowest of the collocated cost values ​​to determine Y collocated candidates. The number Y may be a positive integer.

[0170] In method M2, after the Y collocated blocks are selected, the decoder module 124 may decode the video data based on blocks 860 to 890 of method 800. In this manner, the decoder module 124 may derive SbTMVP candidates based on the motion information of the Y collocated blocks for reconstructing a block unit.

[0171] 10 shows a block diagram of the encoder module 114 of the first electronic device 110 shown in FIG. 1 according to an embodiment of the present disclosure. The encoder module 114 may include a prediction processor (e.g., a prediction process unit 10141), at least a first adder (e.g., a first adder 10142) and a second adder (e.g., a second adder 10145), a transform / quantization processor (e.g., a transform / quantization unit 10143), an inverse quantization / inverse transform processor (e.g., an inverse quantization / inverse transform unit 10144), a filter (e.g., a filtering unit 10146), a decoded picture buffer (e.g., a decoded picture buffer 10147), and an entropy encoder (e.g., an entropy encoding unit 10148). The prediction process unit 10141 of the encoder module 114 may further include a partition processor (e.g., partition unit 101411), an intra-prediction processor (e.g., intra-prediction unit 101412), and an inter-prediction processor (e.g., inter-prediction unit 101413).

[0172] The encoder module 114 may receive a source video, encode the source video, and output a bitstream. The encoder module 114 may receive a source video including a plurality of image frames, and divide the image frames according to a code structure. Each of the image frames may be divided into at least one image block.

[0173] The at least one image block may include a luminance block having a plurality of luminance samples and at least one chrominance block having a plurality of chrominance samples. The luminance block and the at least one chrominance block may be further partitioned to generate a macroblock, a coding tree unit (CTU), a coding block (CB), a subdivision thereof, and / or another equivalent coding unit.

[0174] Encoder module 114 may perform additional sub-division of the source video. It should be noted that the disclosed embodiments are generally applicable to video coding, regardless of how the source video is partitioned prior to and / or during coding.

[0175] During the coding process, the prediction process unit 10141 may receive a current image block of an image frame. The current image block may be a luminance block or a chrominance block of the image frame.

[0176] The partition unit 101411 may divide the current image block into multiple block units. The intra prediction unit 101412 may perform intra predictive coding of the current block unit relative to one or more neighboring blocks in the same frame as the current block unit to provide spatial prediction. The inter prediction unit 101413 may perform inter predictive coding of the current block unit relative to one or more blocks in one or more reference image blocks to provide temporal prediction.

[0177] The prediction process unit 10141 may select one of the loading results generated by the intra prediction unit 101412 and the inter prediction unit 101413 based on a mode selection method such as a cost function. The mode selection method may be a rate-distortion optimization (RDO) process.

[0178] The prediction process unit 10141 may determine a selected coding result and provide a predicted block corresponding to the selected coding result to a first adder 10142 for generating a residual block and a second adder 10145 for reconstructing an encoded block unit. The prediction process unit 10141 may further provide syntax elements, such as motion vectors, intra mode indicators, partition information, and other syntax information, to the entropy encoding unit 10148.

[0179] The intra prediction unit 101412 may intra predict the current block unit. The intra prediction unit 101412 may determine an intra prediction mode directed to reconstructed samples neighboring the current block unit to encode the current block unit.

[0180] The intra prediction unit 101412 may encode the current block unit using various intra prediction modes. The intra prediction unit 101412 of the prediction process unit 10141 may select an appropriate intra prediction mode from the selected modes. The intra prediction unit 101412 may encode the current block unit using a cross-component prediction mode and predict one of the two chroma components of the current block unit based on the luma component of the current block unit. The intra prediction unit 101412 may predict a first of the two chroma components of the current block unit based on a second of the two chroma components of the current block unit.

[0181] The inter prediction unit 101413 may inter predict the current block unit as an alternative to the intra prediction performed by the intra prediction unit 101412. The inter prediction unit 101413 may perform motion estimation to estimate the motion of the current block unit to generate a motion vector.

[0182] The motion vector may indicate the displacement of a current block unit in a current image block relative to a reference block unit in a reference image block. The inter prediction unit 101413 may receive at least one reference image block stored in the decoded picture buffer 10147, and estimate motion based on the received reference image block to generate a motion vector.

[0183] The first adder 10142 may generate a residual block by subtracting the predicted block determined by the prediction process unit 10141 from the original current block unit. The first adder 10142 may represent a component that performs this subtraction.

[0184] The transform / quantization unit 10143 may apply a transform to the residual blocks to generate residual transform coefficients and may quantize the residual transform coefficients to further reduce the bit rate. The transform may be one of a DCT, a DST, an AMT, an MDNSST, a HyGT, a signal-dependent transform, a KLT, a wavelet transform, an integer transform, a sub-band transform, or a conceptually similar transform.

[0185] The transformation may convert the residual information from the pixel value domain to a transform domain, such as the frequency domain. The degree of quantization can be varied by adjusting a quantization parameter.

[0186] The transform / quantization unit 10143 may perform a scan of the matrix containing the quantized transform coefficients, or the entropy encoding unit 10148 may perform the scan.

[0187] The entropy encoding unit 10148 may receive a number of syntax elements, including quantization parameters, transformation data, motion vectors, intra modes, partition information, and other syntax information, from the prediction process unit 10141 and the transform / quantization unit 10143. The entropy encoding unit 10148 may encode the syntax elements into a bitstream.

[0188] An entropy encoding unit 10148 may entropy encode the quantized transform coefficients by performing CAVLC, CABAC, SBAC, PIPE coding, or another entropy coding technique to generate an encoded bitstream that may be transmitted to another device (i.e., the second electronic device 120 of FIG. 1) or archived for later transmission or retrieval.

[0189] An inverse quantization / inverse transform unit 10144 may apply inverse quantization and inverse transform to reconstruct the residual block in the pixel domain for later use as a reference block. A second adder 10145 may add the reconstructed residual block to the prediction block provided from the prediction process unit 10141 to generate a reconstructed block for storage in a decoded picture buffer 10147.

[0190] The filtering unit 10146 may include a deblocking filter, a SAO filter, a bilateral filter, and / or an ALF to remove blocking artifacts from the reconstructed blocks. In addition to the deblocking filter, the SAO filter, the bilateral filter, and the ALF, additional filters (in-loop or post-loop) may be used. Such filters are not shown for simplicity and may filter the output of the second adder 10145.

[0191] The decoded picture buffer 10147 may be a reference picture memory that stores reference blocks used by the encoder module 114 to encode video, such as in intra-coding or inter-coding modes. The decoded picture buffer 10147 may include a variety of memory devices, such as DRAM (including, for example, SDRAM), MRAM, RRAM, or other types of memory devices. The decoded picture buffer 10147 may be on-chip with other components of the encoder module 114 or off-chip relative to those components.

[0192] The method 300 for decoding / encoding video data may be performed by the first electronic device 110. The encoder module 114 may receive video data. The video data received by the encoder module 114 may be a video. The encoder module 114 may determine a block unit from an image according to the video data. The encoder module 114 may divide an image frame to generate a plurality of CTUs, and further divide one of the CTUs to determine a block unit according to one of a plurality of partition schemes based on any video coding standard.

[0193] 1, 3, and 10, the encoder module 114 may determine a plurality of candidate positions including at least one non-adjacent position that is not adjacent to the block unit. The candidate positions may be a plurality of neighboring positions that are adjacent to the block unit. The neighboring positions may be selected from at least one of a plurality of neighboring positions that are adjacent to the block unit or a plurality of non-adjacent positions that are not adjacent to the block unit.

[0194] Referring to FIG. 1, FIG. 3, and FIG. 10, the encoder module 114 may determine a plurality of motion candidates from a plurality of candidate positions. The encoder module 114 may determine a plurality of neighboring blocks adjacent to the block unit based on the candidate positions. Each of the neighboring blocks may cover at least one of the candidate positions. The encoder module 114 may determine a template region adjacent to the block unit from the image frame, and determine a plurality of neighboring regions from the image frame. Each of the neighboring regions may be adjacent to a corresponding one of the neighboring blocks. The encoder module 114 may determine a neighborhood cost value based on the template region and each of the plurality of neighboring regions using a cost function for selecting a motion candidate for the block unit. The encoder module 114 may select a plurality of candidate blocks based on the neighborhood cost values ​​calculated by the neighboring regions. The candidate blocks may be reconstructed blocks reconstructed before reconstructing the block unit. If one of the candidate blocks does not have motion information, a motion candidate may not be determined based on the corresponding one of the selected neighboring positions. Additionally, the motion candidates may further include history-based candidates determined from a first-in-first-out (FIFO) table that stores multiple previous motion information of multiple previous blocks that were reconstructed before reconstructing the block unit.

[0195] Referring to FIG. 1, FIG. 3, and FIG. 10, the encoder module 114 may select a plurality of collocated blocks each included in one of the one or more collocated frames based on a plurality of motion candidates. Each of the plurality of motion candidates may indicate a motion vector, list information, and frame information of a candidate block covering one of a plurality of candidate positions. The encoder module 114 may determine one or more collocated frames based on at least one predetermined value. When one of a plurality of frame indexes in the frame information of a particular one of the candidate blocks is equal to one of the at least one predetermined value, a reference frame of a particular candidate block selected from a corresponding one of the reference lists indicated by the list information of the particular candidate block may be included in one or more collocated frames. The encoder module 114 may determine a motion block using a particular one of the motion candidates corresponding to a particular candidate block when a reference frame selected by the frame information of the particular motion candidate is included in one or more collocated frames. Each motion block may be considered as a collocated block. In some embodiments, the encoder module 114 may determine a motion cost value for each of the motion blocks generated in different collocated frames based on the same motion vector, and the encoder module 114 may compare the motion cost values ​​of the motion blocks to select a collocated block.

[0196] Referring to FIG. 1, FIG. 3, and FIG. 10, the encoder module 114 may determine a collocated cost value for each of the plurality of collocated block units. The encoder module 114 may determine a plurality of collocated regions from one or more collocated frames, each of which is adjacent to one of the plurality of collocated blocks. Each of the collocated regions may include at least one of the plurality of adjacent regions of the corresponding one of the collocated blocks. The encoder module 114 may determine a collocated cost value based on the template region and each of the plurality of collocated regions using a cost function.

[0197] 1, 3, and 10, the encoder module 114 determines at least one sub-block-based temporal motion vector prediction (SbTMVP) candidate based on a plurality of motion candidates and a plurality of first cost values. The encoder module 114 may select at least one of the collocated blocks based on the first cost value calculated by the collocated region. Each of the at least one SbTMVP candidate may be determined based on a corresponding one of the at least one collocated candidate.

[0198] Referring to FIG. 1, FIG. 3, and FIG. 10, the encoder module 114 may select one of a plurality of mode candidates in a candidate list including at least one SbTMVP candidate based on a mode selection method such as a cost function for predicting the block unit. The encoder module 114 may predict the block unit based on at least one SbTMVP candidate in the candidate list to generate a plurality of predicted blocks. Then, the encoder module 114 may further compare a plurality of predicted samples included in each of the predicted blocks with a plurality of color components in the block unit. In this manner, the encoder module 114 may determine a block cost value for each of the predicted blocks and select a prediction candidate for the block unit from the mode candidates in the candidate list based on the block cost value. The mode selection method may be a rate distortion optimization (RDO) process. Then, the encoder module 114 may determine a plurality of residual components by comparing the color components of the block unit with the prediction samples generated based on the prediction mode. Furthermore, the residual components and the prediction index may be encoded into a bitstream for the decoder device to reconstruct the video data. The prediction index may indicate a selected one of the mode candidates in the candidate list. In some embodiments, the prediction index may be an index SbTMVP_idx for determining a selected one of the SbTMVP candidates when the candidate list includes only SbTMVP candidates.

[0199] 1, 3, and 10, the encoder module 114 may further reconstruct the block unit based on the prediction candidates in the candidate list to generate a reconstructed block including a plurality of reconstructed samples. The reconstructed samples of the block unit may be used as references for predicting a plurality of subsequent blocks in the video data.

[0200] Therefore, the encoder module 114 may also use the method 300 to predict and reconstruct block units for encoding video data into a bitstream. Because the encoder module 114 and the decoder module 124 may use the same method 300, the encoder module 114 and the decoder module 124 may generate the same reordering list to reduce the number of bits in the bitstream.

[0201] A method M1 similar to the method 300 for decoding / encoding video data may be executed by the first electronic device 110. In the method M1, the encoder module 114 may encode the video data based on the method 300. Further, in block 370, after selecting at least one of the collocated blocks, the encoder module 114 further determines each of a plurality of shift blocks adjacent to one of the selected at least one collocated block. In some embodiments, each of the shift blocks may be selected based on a plurality of neighboring positions, including a plurality of adjacent positions and a plurality of non-adjacent positions in FIG. 4.

[0202] 1, 3, and 10, the encoder module 114 may determine a shift cost value based on the template region and each of the multiple collocated regions by using a cost function. In FIG. 7, the encoder module 114 may directly compare the template region 7000 of the block unit 700 with each of the collocated regions 7700-7740 by the cost function. If one of the shift cost values ​​determined based on the collocated region of the shift block is smaller than the shift cost value determined based on the collocated region of the collocated block, the corresponding one of the shift blocks may replace the collocated block to determine the collocated candidate.

[0203] The method 800 for decoding / encoding video data may be performed by the first electronic device 110. The encoder module 114 may receive video data. The video data received by the encoder module 114 may be a video. The encoder module 114 may determine a block unit from an image according to the video data. The encoder module 114 may divide an image frame to generate a plurality of CTUs, and further divide one of the CTUs to determine a block unit according to one of a plurality of division schemes based on any video coding standard.

[0204] 1, 8, and 10, the encoder module 114 may determine a plurality of candidate positions selected from a plurality of adjacent positions adjacent to the block unit and a plurality of non-adjacent positions not adjacent to the block unit. The candidate positions may be a plurality of nearby positions adjacent to the block unit. The nearby positions may be selected from at least one of a plurality of adjacent positions adjacent to the block unit or a plurality of non-adjacent positions not adjacent to the block unit.

[0205] Referring to FIG. 1, FIG. 8, and FIG. 10, the encoder module 114 may determine a plurality of motion candidates from a plurality of candidate positions. The encoder module 114 may determine a plurality of neighboring blocks adjacent to the block unit based on the candidate positions. Each of the neighboring blocks may cover at least one of the candidate positions. The encoder module 114 may determine a template region adjacent to the block unit from the image frame and determine a plurality of neighboring regions from the image frame. Each of the neighboring regions may be adjacent to a corresponding one of the neighboring blocks. The encoder module 114 may determine a neighborhood cost value based on the template region and each of the plurality of neighboring regions using a cost function to select a motion candidate for the block unit. The encoder module 114 may select a plurality of candidate blocks based on the neighborhood cost value calculated by the neighboring regions. The candidate blocks may be reconstructed blocks that are reconstructed before reconstructing the block unit. If one of the candidate blocks does not have motion information, a motion candidate may not be determined based on the corresponding one of the selected neighboring positions. Additionally, the motion candidates may further include history-based candidates determined from a first-in-first-out (FIFO) table that stores multiple previous motion information of multiple previous blocks that were reconstructed prior to reconstructing the block unit.

[0206] Referring to FIG. 1, FIG. 8, and FIG. 10, the encoder module 114 may determine a plurality of collocated blocks, each of which is included in one of the one or more collocated frames, based on a plurality of motion candidates. Each of the plurality of motion candidates may indicate a motion vector, list information, and frame information of a candidate block covering one of a plurality of candidate positions. The encoder module 114 may determine one or more collocated frames based on at least one predetermined value. When one of a plurality of frame indexes of the frame information of a particular one of the candidate blocks is equal to one of the at least one predetermined value, a reference frame of the particular candidate block selected from a corresponding one of the reference lists indicated by the list information of the particular candidate block may be included in one or more collocated frames. The encoder module 114 may determine a motion block using a particular one of the motion candidates corresponding to the particular candidate block when a reference frame selected by the frame information of the particular motion candidate is included in one or more collocated frames. Each motion block may be considered as a collocated block. In some embodiments, the encoder module 114 may determine a motion cost value for each of the motion blocks generated in different collocated frames based on the same motion vector, and the encoder module 114 may compare the motion cost values ​​of the motion blocks to select a collocated block.

[0207] Referring to FIG. 1, FIG. 8, and FIG. 10, the encoder module 114 may derive a plurality of sub-block-based temporal motion vector prediction (SbTMVP) candidates based on the motion information of a plurality of collocated blocks. The encoder module 114 may determine whether the collocated blocks have motion information. The encoder module 114 may use the motion information of the collocated blocks to determine SbTMVP candidates that respectively correspond to one of the collocated blocks. When a block unit is predicted based on an SbTMVP candidate, the block unit may be divided into a plurality of sub-block units. Thus, each of the collocated blocks may also be divided into a plurality of collocated sub-blocks. Before reconstructing an image frame, the encoder module 114 may use the motion information of the collocated blocks to determine a plurality of first sub-block vectors Vs1 for the collocated sub-blocks since the collocated frame is reconstructed. Then, the encoder module 114 may derive a plurality of second subblock vectors Vs2 for the subblock unit based on the plurality of first subblock vectors Vs1. Furthermore, when a particular one of the collocated subblocks is included in a uni-predictive block, the encoder module 114 may derive one second subblock vector for a particular one of the subblock units corresponding to the particular collocated subblock. Furthermore, when a particular collocated subblock is included in a bi-predictive block, the encoder module 114 may derive two second subblock vectors for a particular subblock unit corresponding to the particular collocated subblock.

[0208] Referring to FIG. 1, FIG. 8, and FIG. 10, the encoder module 114 may determine a reference cost value based on each of the multiple SbTMVP candidates. The encoder module 114 may determine a reference region including multiple reference sub-regions each generated for a reference sub-block corresponding to one of the collocated blocks. Since the reference sub-blocks corresponding to a specific one of the collocated blocks may not be adjacent to each other, the reference sub-region corresponding to the specific collocated block may be determined based on a portion of the M second sub-block vectors Vs2. The portion of the M second sub-block vectors Vs2 may belong to a sub-block unit located at the upper boundary of the block unit or a sub-block unit located at the left boundary of the block unit. The encoder module 114 may determine a reference cost value based on the template region and each of the multiple reference regions using a cost function.

[0209] 1, 8, and 10, the encoder module 114 may determine an arrangement of the SbTMVP candidates based on the first cost value, and generate an adjusted SbTMVP list based on the arrangement. The encoder module 114 may determine an arrangement of the SbTMVP candidates based on the reference cost value, and rearrange the SbTMVP candidates based on the arrangement. The arrangement may further be determined by adjusting the new order of the SbTMVP candidates based on a diversity criterion. In some embodiments, the SbTMVP candidates may be divided into multiple candidate groups based on multiple division criteria. For example, the SbTMVP candidates may be divided into multiple candidate groups based on a neighborhood position type, a block size of a block unit, or at least one predetermined number. Furthermore, the encoder module 114 may collect only P SbTMVP candidates, and the amount of P SbTMVP candidates may be sufficient to distribute to the first p-th candidate group.

[0210] Referring to FIG. 1, FIG. 8, and FIG. 10, the encoder module 114 may select one of a plurality of mode candidates in a candidate list including at least one SbTMVP candidate based on a mode selection method such as a cost function to predict the block unit. The encoder module 114 may predict the block unit based on at least one SbTMVP candidate in the candidate list to generate a plurality of predicted blocks. Then, the encoder module 114 may further compare a plurality of predicted samples included in each of the predicted blocks with a plurality of color components in the block unit. In this manner, the encoder module 114 may determine a block cost value for each of the predicted blocks and select a prediction candidate for the block unit from the mode candidates in the candidate list based on the block cost value. The mode selection method may be a rate distortion optimization (RDO) process. Then, the encoder module 114 may determine a plurality of residual components by comparing the color components of the block unit with the prediction samples generated based on the prediction mode. Furthermore, the residual components and the prediction index may be encoded into a bitstream for the decoder device to reconstruct the video data. The prediction index may indicate a selected one of the mode candidates in the candidate list. In some embodiments, the prediction index may be an index SbTMVP_idx for determining a selected one of the SbTMVP candidates when the candidate list includes only SbTMVP candidates.

[0211] 1, 8, and 10, the encoder module 114 may further reconstruct the block unit based on the prediction candidates in the candidate list to generate a reconstructed block including a plurality of reconstructed samples. The reconstructed samples of the block unit may be used as references for predicting a plurality of subsequent blocks in the video data.

[0212] Accordingly, the encoder module 114 may also use the method 800 to predict and reconstruct block units for encoding video data into a bitstream. Because the encoder module 114 and the decoder module 124 may use the same method 800, the encoder module 114 and the decoder module 124 may generate the same reordering list to reduce the number of bits in the bitstream.

[0213] A method M2, which is generated by combining the method 300 and the method 800, which are methods for encoding / decoding video data, may be executed by the first electronic device 110. In the method M2, the encoder module 114 may encode the video data based on the blocks 310-360 of the method 300. Then, the encoder module 114 may select a portion of the collocated block based on the collocated cost value of the collocated block. After the portion of the collocated block is selected, the encoder module 114 may encode the video data based on the blocks 860-890 of the method 800. In this manner, the encoder module 114 may derive SbTMVP candidates based on the motion information of the portion of the collocated block to reconstruct a block unit.

[0214] The disclosed embodiments are considered in all respects to be illustrative and not restrictive, and it is to be understood that the disclosure is not limited to the particular disclosed embodiments, but that many rearrangements, modifications, and substitutions are possible without departing from the scope of the disclosure. [Brief description of the drawings]

[0215] Aspects of the present disclosure are best understood from the following detailed disclosure and corresponding figures, in which: Various features are not drawn to scale and dimensions of various features may be arbitrarily increased or decreased for clarity of discussion. [Figure 1]FIG. 1 is a block diagram of a system configured to encode and decode video data according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a block diagram of a decoder module of the second electronic device illustrated in FIG. 1 according to an embodiment of the present disclosure. [Diagram 3] FIG. 3 shows a flowchart of a method for decoding video data by an electronic device according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram illustrating an example implementation of a block and multiple candidate locations according to one embodiment of the present disclosure. [Diagram 5] 5A, 5B, and 5C are diagrams illustrating an example implementation of a template region and different adjacent regions according to one embodiment of the present disclosure. [Figure 6] FIG. 6 is a diagram illustrating an example implementation of an image frame having block units and template regions, and a collocated frame having collocated blocks and collocated regions, according to one embodiment of the present disclosure. [Figure 7] FIG. 7 is a diagram illustrating an example implementation of an image frame having block units and template regions, and a collocated frame having collocated blocks, shifted blocks, and collocated regions, according to one embodiment of the present disclosure. [Figure 8] FIG. 8 shows a flowchart of a method for decoding video data by an electronic device according to an embodiment of the present disclosure. [Figure 9] 9A and 9B are diagrams illustrating an example implementation of a template region and a reference region according to an embodiment of the present disclosure. [Figure 10] FIG. 10 is a block diagram of an encoder module of the first electronic device shown in FIG. 1 according to an embodiment of the present disclosure.

Claims

1. 1. A method for decoding video data by an electronic device, comprising: receiving the video data; determining block units from an image frame according to the video data; determining a plurality of candidate locations including at least one non-adjacent location that is not adjacent to the block unit; determining a plurality of motion candidates from the plurality of candidate positions; selecting a number of collocated blocks, each of which is comprised in one of one or more collocated frames comprised in the video data, based on the number of motion candidates; determining a first cost value for each of the plurality of collocated blocks; determining at least one sub-block-based temporal motion vector prediction (SbTMVP) candidate based on the plurality of motion candidates and the plurality of first cost values; and reconstructing the block unit based on the at least one SbTMVP candidate.

2. The method of claim 1 , wherein the plurality of candidate locations further includes at least one of a plurality of adjacent locations adjacent to the block unit.

3. determining a plurality of neighboring blocks from the image frame that are neighboring the block unit based on the plurality of candidate locations, where at least one of the plurality of neighboring blocks covers one of the at least one non-neighboring locations; determining, from the image frame, a template region adjacent to the block unit and a plurality of neighboring regions each adjacent to one of the plurality of neighboring blocks; determining a second cost value based on the template region and each of the plurality of neighboring regions; selecting a plurality of candidate blocks based on the plurality of second cost values; and determining the motion candidates from the candidate blocks; The method of claim 1 further comprising:

4. each of the plurality of motion candidates indicates a motion vector, list information, and frame information of a candidate block covering one of the plurality of candidate positions; 2. The method of claim 1, wherein if a frame index in the frame information of a particular candidate block is equal to one of at least one predetermined value, a reference frame of the particular candidate block selected from a reference list indicated by list information of the particular candidate block is included in the one or more collocated frames.

5. The method of claim 4 , wherein one of the plurality of collocated blocks is selected from the reference frame for the particular candidate block based on the motion vector and a corresponding one of the plurality of candidate locations.

6. The method of claim 4 , wherein the reference frame of the particular candidate block is excluded from the one or more collocated frames if the frame index in the frame information of the particular candidate block differs from at least one predetermined value.

7. each of the plurality of motion candidates indicates a motion vector, list information, and frame information of a candidate block covering one of the plurality of candidate positions; If the particular candidate block is a bi-predictive block, at least one reference frame for the particular candidate block is determined from two reference lists for the particular candidate block; 2. The method of claim 1, wherein the at least one reference frame is included in the one or more collocated frames if one of two frame indexes included in the frame information of the particular candidate block is equal to one of at least one predetermined value.

8. adding said at least one SbTMVP candidate to a candidate list; selecting a predicted candidate from the candidate list based on a prediction index; and reconstructing the block unit based on a prediction mode; The method of claim 1 further comprising:

9. The method of claim 1 , wherein the plurality of collocated blocks is selected from a plurality of collocated frames.

10. determining a template region from the image frame adjacent to the block unit, and determining a plurality of collocated regions from the one or more collocated frames, each of the collocated regions being adjacent to one of the plurality of collocated blocks; and determining the first cost value between the template region and each of the plurality of collocated regions; The method of claim 1 further comprising:

11. 1. An electronic device for decoding video data, comprising: At least one processor; coupled to the at least one processor and, when executed by the at least one processor, causing the electronic device to receiving the video data; determining a block unit from an image frame according to the video data; determining a plurality of candidate locations including at least one non-adjacent location that is not adjacent to the block unit; determining a plurality of motion candidates from the plurality of candidate positions; selecting a plurality of collocated blocks, each of which is included in one of one or more collocated frames included in the video data, based on the plurality of motion candidates; determining a first cost value for each of the plurality of collocated blocks; determining at least one sub-block based temporal motion vector prediction (SbTMVP) candidate based on the plurality of motion candidates and the plurality of first cost values; and at least one storage device storing a plurality of computer-executable instructions to cause the electronic apparatus to perform the steps of: reconstructing the block unit based on the at least one SbTMVP candidate.

12. The electronic device of claim 10 , wherein the plurality of candidate locations further includes at least one of a plurality of adjacent locations adjacent to the block unit.

13. The computer-executable instructions, when executed by the at least one processor, further cause the electronic device to: determining a plurality of neighboring blocks adjacent to the block unit from the image frame based on the plurality of candidate locations; determining a template region adjacent to the block unit and a plurality of neighboring regions from the image frame, each neighboring one of the plurality of neighboring blocks; determining a second cost value based on the template region and each of the plurality of neighboring regions; selecting a plurality of candidate blocks based on the plurality of second cost values; determining the plurality of motion candidates from the plurality of candidate blocks; The electronic device of claim 10 , wherein at least one of the plurality of neighboring blocks covers one of the at least one non-adjacent location.

14. each of the plurality of motion candidates indicates a motion vector, list information, and frame information of a candidate block covering one of the plurality of candidate positions; 11. The electronic device of claim 10, wherein, if a frame index in the frame information of a particular candidate block is equal to one of at least one predetermined value, a reference frame of the particular one candidate block selected from a reference list indicated by list information of the particular candidate block is included in the one or more collocated frames.

15. The electronic device of claim 14 , wherein one of the plurality of collocated blocks is selected from the reference frame for the particular candidate block based on the motion vector and a corresponding one of the plurality of candidate locations.

16. 15. The electronic device of claim 14, wherein the reference frame of the particular candidate block is excluded from the one or more collocated frames if the frame index in the frame information of the particular candidate block differs from at least one predetermined value.

17. each of the plurality of motion candidates indicates a motion vector, list information, and frame information of a candidate block covering one of the plurality of candidate positions; if the particular candidate block is a bi-predictive block, at least one reference frame for the particular one of the candidate blocks is determined from two reference lists for the particular candidate block; 11. The electronic device of claim 10, wherein if one of two frame indexes included in the frame information of the particular candidate block is equal to one of at least one predetermined value, the at least one reference frame is included in the one or more collocated frames.

18. The computer-executable instructions, when executed by the at least one processor, further cause the electronic device to: adding said at least one SbTMVP candidate to a candidate list; selecting a prediction candidate from the candidate list based on a prediction index; The electronic device of claim 10 , configured to reconstruct the block units based on a prediction mode.

19. The electronic device of claim 10 , wherein the plurality of collocated blocks are selected from a plurality of collocated frames.

20. The computer-executable instructions, when executed by the at least one processor, further cause the electronic device to: determining a template region from the image frame adjacent to the block unit; and determining a plurality of collocated regions from the one or more collocated frames, each of the collocated regions being adjacent to one of the plurality of collocated blocks; The electronic device of claim 10 , further configured to determine the first cost value between the template region and each of the plurality of collocated regions.

Images