Apparatus and method for decoding video data

Abstract

A method for decoding video data by an electronic device is provided. The electronic device receives video data and determines a block unit from an image frame according to the video data. The electronic device further determines a plurality of candidate positions including at least one non-adjacent position not adjacent to the block unit, and determines a plurality of motion candidates from the candidate positions. The electronic device further selects a plurality of collocated blocks each included in one of one or more collocated frames (collocated blocks) included in the video data based on the motion candidates, and determines a first cost value for each of the collocated blocks. Then, the electronic device determines at least one sub-block-based temporal motion vector prediction (SbTMVP) candidate based on the motion candidates and the first cost value, and reconstructs the block unit based on the at least one SbTMVP candidate.

Description

Detailed Description of the Invention

[0001] [Technical Field] This disclosure claims the benefit of and priority to U.S. Provisional Patent Application Serial No. 63 / 265442, filed December 15, 2021, entitled "TEMPORAL-BASED PREDICTION TOOLS," and U.S. Provisional Patent Application Serial No. 63 / 265777, filed December 20, 2021, entitled "SUBBLOCK TMVP IMPROVEMENT BASED ON TEMPLATE MATCHING," the contents of which are incorporated by reference in their entireties into this disclosure.

[0002] TECHNICAL FIELD This disclosure relates generally to video decoding, and more particularly to techniques using template prediction.

[0003] [Background technology] Subblock-based temporal motion vector prediction (SbTMVP) is a coding tool for video coding. In conventional video coding methods, an encoder and a decoder can determine a collocated block in a collocated frame for a candidate block in an image frame and generate one of multiple SbTMVP candidates for predicting the candidate block.

[0004] The encoder and decoder may divide the collocated block into multiple collocated sub-blocks and determine multiple first sub-block vectors Vs1 for the collocated sub-blocks within the collocated block using motion information of the collocated block. Then, the encoder and decoder may divide the candidate block into multiple target sub-blocks and derive multiple second sub-block vectors Vs2 for the target sub-blocks within the block unit based on the first sub-block vectors Vs1 of the collocated sub-blocks. Each of the second sub-block vectors Vs2 may be derived based on a corresponding one of the first sub-block vectors Vs1, a reference distance between the image frame and the reference frame of the block unit, and a collocated distance between the collocated frame and the collocated reference frame of the collocated block.

[0005] However, because the SbTMVP candidates for a candidate block are only arranged based on the collection order, when predicting a candidate block using SbTMVP candidates, the coding efficiency for signaling the index for determining the selected SbTMVP candidate may not be high enough. Therefore, the encoder and decoder may need a new SbTMVP to more accurately predict or reconstruct the candidate block. 〔overview〕

[0006] The present disclosure is directed to an apparatus and method for predicting block units in an image frame using template prediction.

[0007] In a first aspect of the present disclosure, there is provided a method for decoding video data and an electronic device for performing the method, the method including: receiving video data; determining a block unit from an image frame according to the video data; determining a plurality of candidate locations including at least one non-adjacent location that is not adjacent to the block unit; determining a plurality of motion candidates from the plurality of candidate locations; selecting a plurality of collocated blocks, each of which is included in one of one or more collocated frames included in the video data, based on the plurality of motion candidates; determining a first cost value for each of the plurality of collocated blocks; determining at least one sub-block-based temporal motion vector prediction (SbTMVP) candidate based on the plurality of motion candidates and the plurality of first cost values; and reconstructing a block unit based on the at least one SbTMVP candidate.

[0008] In another embodiment of the first aspect, the plurality of candidate locations further includes at least one of a plurality of adjacent locations adjacent to the block unit.

[0009] An embodiment of the first aspect further includes: determining, from the image frame, a plurality of neighboring blocks adjacent to the block unit based on the plurality of candidate positions, wherein at least one of the plurality of neighboring blocks covers one of the at least one non-adjacent positions; determining, from the image frame, a template region adjacent to the block unit and a plurality of neighboring regions, each neighboring region being adjacent to one of the plurality of neighboring blocks; determining a second cost value based on the template region and each of the plurality of neighboring regions; selecting a plurality of candidate blocks based on the plurality of second cost values; and determining the plurality of motion candidates from the plurality of candidate blocks.

[0010] In another embodiment of the first aspect, each of the plurality of motion candidates indicates a motion vector, list information, and frame information of a candidate block covering one of the plurality of candidate positions, and if a frame index in the frame information of a particular candidate block is equal to one of at least one predetermined value, the reference frame of a particular one of the candidate blocks selected from the reference list indicated by the list information of the particular candidate block is included in one or more collocated frames.

[0011] In another embodiment of the first aspect, one of the plurality of collocated blocks is selected from the reference frame of the particular candidate block based on the motion vector and a corresponding one of the plurality of candidate locations.

[0012] In another embodiment of the first aspect, if a frame index of the frame information of the particular candidate block is different from at least one predetermined value, the reference frame of the particular candidate block is excluded from the one or more collocated frames.

[0013] In another embodiment of the first aspect, each of the plurality of motion candidates indicates a motion vector, list information, and frame information of a candidate block covering one of the plurality of candidate positions, and at least one reference frame for a particular one of the candidate blocks is determined from two reference lists for the particular candidate block if the particular candidate block is a bi-predictive block, and the at least one reference frame is included in the one or more collocated frames if one of two frame indexes in the frame information of the particular candidate block is equal to one of at least one predetermined value.

[0014] An embodiment of the first aspect further includes adding the at least one SbTMVP candidate to a candidate list, selecting a prediction candidate from the candidate list based on a prediction index, and reconstructing the block unit based on the prediction mode.

[0015] In another embodiment of the first aspect, the plurality of collocated blocks are selected from a plurality of collocated frames.

[0016] An embodiment of the first aspect further includes the steps of: determining a template region adjacent to the block unit from the image frame; determining a plurality of collocated regions from one or more collocated frames, each of which is adjacent to one of a plurality of collocated blocks; and determining the first cost value between the template region and each of the plurality of collocated regions.

[0017] BRIEF DESCRIPTION OF THE DRAWINGS Aspects of the present disclosure are best understood from the following detailed disclosure and corresponding figures, in which: Various features are not drawn to scale and dimensions of various features may be arbitrarily increased or decreased for clarity of discussion. FIG. 1 is a block diagram of a system configured to encode and decode video data according to an embodiment of the present disclosure. 2 is a block diagram of a decoder module of the second electronic device illustrated in FIG. 1 according to an embodiment of the present disclosure. FIG. 3 shows a flowchart of a method for decoding video data by an electronic device according to an embodiment of the present disclosure. FIG. 4 is an explanatory diagram illustrating an exemplary embodiment of a block and a plurality of candidate locations according to an embodiment of the present disclosure. 5A, 5B, and 5C are diagrams illustrating an exemplary implementation of a template region and different neighborhood regions according to one embodiment of the present disclosure. FIG. 6 is an explanatory diagram showing an example implementation of an image frame having block units and template regions, and a collocated frame having collocated blocks and collocated regions, according to an embodiment of the present disclosure. FIG. 7 is an explanatory diagram showing an example implementation of an image frame having a block unit and a template region, and a collocated frame having a collocated block, a shift block, and a collocated region, according to one embodiment of the present disclosure. FIG. 8 shows a flowchart of a method for decoding video data by an electronic device according to an embodiment of the present disclosure. 9A and 9B are diagrams illustrating exemplary embodiments of a template region and a reference region according to an embodiment of the present disclosure. 10 is a block diagram of an encoder module of the first electronic device shown in FIG. 1 according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0018] The following disclosure contains specific information relevant to the implementation of the present disclosure. The figures and corresponding detailed disclosure are directed to exemplary embodiments. However, the present disclosure is not limited to these exemplary embodiments. Other variations and exemplary embodiments of the present disclosure will occur to those skilled in the art.

[0019] Unless otherwise noted, like or corresponding elements in the figures may be indicated with like or corresponding reference designators. The figures and illustrations in this disclosure are generally not to scale and are not intended to correspond to actual relative dimensions.

[0020] For purposes of consistency and ease of understanding, like features are identified in the exemplary figures by reference designators (although not shown in some exemplary embodiments). However, features in different embodiments may differ in other respects and are not intended to be narrowly limited to those illustrated in the figures.

[0021] In this disclosure, the phrases "in one embodiment" or "in some embodiments" may refer to one or more of the same or different embodiments. The term "coupled" means connected through intervening components, whether direct or indirect, and not necessarily limited to a physical connection. The term "comprising" means "including, but not necessarily limited to," and specifically indicates an open-ended inclusion or membership in such stated combinations, groups, series, and the like.

[0022] For purposes of explanation and not limitation, specific details such as functional entities, techniques, protocols, and standards are set forth to provide an understanding of the disclosed technology. Detailed disclosure of well-known methods, techniques, systems, and architectures is omitted so as not to obscure the present disclosure with unnecessary detail.

[0023] Those skilled in the art will recognize that the disclosed code function(s) or algorithm(s) described in this disclosure may be implemented by hardware, software, or a combination of software and hardware. The disclosed functions may correspond to modules that are software, hardware, firmware, or a combination thereof.

[0024] A software implementation may include a program having computer-executable instructions stored on a computer-readable medium, such as a memory or other type of storage device. For example, one or more microprocessors or general-purpose computers with communications processing capabilities can be programmed with the executable instructions to perform the disclosed function(s) or algorithm(s).

[0025] A microprocessor or general-purpose computer can be formed using an application-specific integrated circuit (ASIC), a programmable logic array, and / or one or more digital signal processors (DSPs). While some of the disclosed implementations are directed to software installed and executed on computer hardware, alternative implementations implemented as firmware, hardware, or a combination of hardware and software are within the scope of this disclosure. Computer-readable media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, compact disc read-only memory (CD-ROM), magnetic cassette, magnetic tape, magnetic disk storage, or any other equivalent medium capable of storing computer-readable instructions.

[0026] 1 shows a block diagram of a system 100 configured to encode and decode video data according to an embodiment of the present disclosure. The system 100 includes a first electronic device 110, a second electronic device 120, and a communication medium 130.

[0027] First electronic device 110 may be a source device, which may include any device configured to encode video data and transmit the encoded video data over communication medium 130. Second electronic device 120 may be a destination device, which may include any device configured to receive the encoded video data over communication medium 130 and decode the encoded video data.

[0028] First electronic device 110 can communicate with second electronic device 120 via a communication medium 130, either wired or wirelessly. First electronic device 110 may include a source module 112, an encoder module 114, and a first interface 116. Second electronic device 120 may include a display module 122, a decoder module 124, and a second interface 126. First electronic device 110 may be a video encoder, and second electronic device 120 may be a video decoder.

[0029] First electronic device 110 and / or second electronic device 120 may be a mobile phone, tablet, desktop, notebook, or other electronic device. Figure 1 shows an example of first electronic device 110 and second electronic device 120. First electronic device 110 and second electronic device 120 may include more or fewer components than shown or may have a different configuration of the various components shown.

[0030] The source module 112 may include a video capture device for capturing new video, a video archive for storing previously captured video, and / or a video feed interface for receiving video from a video content provider. The source module 112 may generate computer graphics-based data as the source video, or may generate a combination of live video, archived video, and computer-generated video as the source video. The video capture device may be a charge-coupled device (CCD) image sensor, a complementary metal-oxide semiconductor (CMOS) image sensor, or a camera.

[0031] The encoder module 114 and the decoder module 124 may each be implemented as any of a variety of suitable encoder / decoder circuits, such as one or more microprocessors, central processing units (CPUs), graphics processing units (GPUs), systems-on-chips (SoCs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. If implemented partially in software, the device may store a program having software instructions on a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the disclosed methods. Each of the encoder module 114 and the decoder module 124 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder / decoder (CODEC) within the device.

[0032] The first interface 116 and the second interface 126 may conform to a customized protocol or to an existing or de facto standard, including, but not limited to, Ethernet, the IEEE 802.11 or IEEE 802.15 series, wireless USB, or a telecommunications standard, including, but not limited to, Global System for Mobile Communications (GSM), Code-Division Multiple Access 2000 (CDMA2000), Time Division Synchronous Code Division Multiple Access (TD-SCDMA), Worldwide Interoperability for Microwave Access (WiMAX), Third Generation Partnership Project Long-Term Evolution (3GPP-LTE), or Time-Division LTE (TD-LTE). The first interface 116 and the second interface 126 may each include any device configured to transmit and / or store compliant video bitstreams over a communication medium 130 and receive compliant video bitstreams over the communication medium 130.

[0033] First interface 116 and second interface 126 may include computer system interfaces that allow compliant video bitstreams to be stored on or received from a storage device. For example, first interface 116 and second interface 126 may include chipsets that support Peripheral Component Interconnect (PCI) and Peripheral Component Interconnect Express (PCIe) bus protocols, proprietary bus protocols, Universal Serial Bus (USB) protocols, Inter-Integrated Circuit (I2C) protocols, or any other logical and physical structure that may be used to interconnect peer devices.

[0034] The display module 122 may include a display using liquid crystal display (LCD), plasma display, organic light emitting diode (OLED), or light emitting polymer display (LPD) technology, as well as other display technologies used in other implementations. The display module 122 may include a high-definition display or an ultra-high-definition display.

[0035] 2 is a block diagram of the decoder module 124 of the second electronic device 120 shown in FIG. 1 according to an embodiment of the present disclosure. The decoder module 124 includes an entropy decoder (e.g., entropy decoding unit 2241), a prediction processor (e.g., prediction processing unit 2242), an inverse quantization / inverse transform processor (e.g., inverse quantization / inverse transform unit 2243), an adder (e.g., adder 2244), a filter (e.g., filtering unit 2245), and a decoded picture buffer (e.g., decoded picture buffer 2246). The prediction processing unit 2242 further includes an intra-prediction processing unit (e.g., intra-prediction unit 22421) and an inter-prediction processing unit (e.g., inter-prediction unit 22422). The decoder module 124 receives a bitstream, decodes the bitstream, and outputs decoded video.

[0036] 1, and may perform a parsing operation on the bitstream to extract the syntax elements from the bitstream. As part of the parsing operation, the entropy decoding unit 2241 may entropy decode the bitstream to generate quantized transform coefficients, quantization parameters, transform data, motion vectors, intra modes, partition information, and other syntax information.

[0037] The entropy decoding unit 2241 may perform context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding technique to generate quantized transform coefficients. The entropy decoding unit 2241 may provide the quantized transform coefficients, quantization parameters, and transform data to the inverse quantization / inverse transform unit 2243 and provide motion vectors, intra modes, partition information, and other syntax information to the prediction process unit 2242.

[0038] The prediction process unit 2242 may receive syntax elements such as motion vectors, intra modes, partition information, and other syntax information from the entropy decoding unit 2241. The prediction process unit 2242 may receive syntax elements including the partition information and divide the image frame according to the partition information.

[0039] Each image frame may be divided into at least one image block according to the partition information. The at least one image block may include a luminance block for reconstructing a plurality of luminance samples and at least one chrominance block for reconstructing a plurality of chrominance samples. The luminance block and the at least one chrominance block may be further divided to generate a macroblock, a coding tree unit (CTU), a coding block (CB), a subdivision thereof, and / or another equivalent coding unit.

[0040] During the decoding process, the prediction process unit 2242 may receive prediction data including intra-mode or motion vectors for a particular current image block of an image frame, which may be one of the luminance or chrominance blocks of the particular image frame.

[0041] The intra prediction unit 22421 may perform intra prediction encoding of the current block unit relative to one or more neighboring blocks in the same frame as the current block unit based on a syntax element associated with an intra mode to generate a predicted block. The intra mode may specify the location of a reference sample selected from the neighboring block in the current frame. The intra prediction unit 22421 may reconstruct multiple chroma components of the current block unit based on multiple luma components of the current block unit when the chroma components are reconstructed by the prediction process unit 2242.

[0042] The intra prediction unit 22421 may reconstruct multiple chroma components of the current block unit based on the multiple luma components of the current block unit when the luma component of the current block is reconstructed by the prediction process unit 2242.

[0043] The inter prediction unit 22422 may perform inter prediction encoding of the current block unit relative to one or more blocks in one or more reference image blocks based on syntax elements related to a motion vector to generate a predicted block. The motion vector may indicate the displacement of the current block unit in the current image block relative to a reference block unit in the reference image block. The reference block unit is a block determined to closely match the current block unit. The inter prediction unit 22422 may receive reference image blocks stored in the decoded picture buffer 2246 and reconstruct the current block unit based on the received reference image blocks.

[0044] The inverse quantization / inverse transform unit 2243 may apply inverse quantization and an inverse transform to reconstruct the residual block in the pixel domain. The inverse quantization / inverse transform unit 2243 may apply inverse quantization to the residual quantized transform coefficients to generate residual transform coefficients, and apply an inverse transform to the residual transform coefficients to generate the residual block in the pixel domain.

[0045] The inverse transform may be applied by a transformation process such as the discrete cosine transform (DCT), discrete sine transform (DST), adaptive multi-layer transform (AMT), mode-dependent non-separable quadratic transform (MDNSST), hypercube-givens transform (HyGT), signal-dependent transform, Karhunen-Loewe transform (KLT), wavelet transform, integer transform, subband transform, or a conceptually similar transform. The inverse transform can return residual information from a transform domain, such as the frequency domain, back to the pixel domain. The degree of inverse quantization can be changed by adjusting the quantization parameter.

[0046] The adder 2244 adds the reconstructed residual block to the prediction block provided by the prediction process unit 2242 to generate a reconstructed block.

[0047] The filtering unit 2245 may include a deblocking filter, a sample adaptive offset (SAO) filter, a bilateral filter, and / or an adaptive loop filter (ALF) to remove blocking artifacts from the reconstructed blocks. In addition to the deblocking filter, the SAO filter, the bilateral filter, and the ALF, additional filters (in-loop or post-loop) may also be used. Such filters, although not explicitly illustrated for simplicity, may filter the output of the summer 2244. After the filtering unit 2245 performs the filtering process on the reconstructed blocks of a particular image frame, the filtering unit 2245 may output the decoded video to the display module 122 or other video receiving unit.

[0048] The decoded picture buffer 2246 may be a reference picture memory that stores reference blocks used by the prediction processing unit 2242 in decoding the bitstream (in inter-coding mode). The decoded picture buffer 2246 may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The decoded picture buffer 2246 may be on-chip with other components of the decoder module 124 or off-chip relative to those components.

[0049] 3 shows a flowchart of a method 300 for decoding video data by an electronic device according to an embodiment of the present disclosure. The method 300 is merely exemplary, as there are various ways to perform the decoding of video data.

[0050] Method 300 may be performed using the configurations illustrated in Figures 1 and 2, the various elements of which are referenced with respect to method 300. Each block illustrated in Figure 3 may represent one or more processes, methods, or subroutines that are performed.

[0051] The order of the blocks in Figure 3 is exemplary and may be changed, and additional blocks may be added or fewer blocks may be utilized without departing from this disclosure.

[0052] In block 310, the decoder module 124 receives the video data. The video data received by the decoder module 124 may be a bitstream.

[0053] 1 and 2, the second electronic device 120 may receive a bitstream from an encoder, such as the first electronic device 110, or another video provider, via a second interface 126. The second interface 126 may provide the bitstream to a decoder module 124.

[0054] The entropy decoding unit 2241 may decode the bitstream to determine prediction instructions and partition instructions for the image frames. The decoder module 124 may then further reconstruct the image frames based on the prediction instructions and partition instructions. The prediction instructions and partition instructions may include flags and indexes.

[0055] In block 320, the decoder module 124 determines block units from the image frame according to the video data.

[0056] 1 and 2, the decoder module 124 may determine an image frame based on a bitstream, divide the image frame, and determine block units according to a partition instruction in the bitstream. For example, the decoder module 124 may divide the image frame to generate multiple CTUs, and further divide one of the CTUs to determine block units according to a partition instruction based on any video encoding standard.

[0057] In block 330, the decoder module 124 determines a plurality of candidate locations, including at least one non-adjacent location that is not adjacent to the block unit.

[0058] The candidate locations may be a plurality of nearby locations adjacent to the block unit. The nearby locations may be selected from at least one of a plurality of nearby locations adjacent to the block unit or a plurality of non-adjacent locations not adjacent to the block unit. Thus, in some embodiments, the candidate locations may include at least one of adjacent locations or non-adjacent locations. In some embodiments, the candidate locations may include at least one of adjacent locations and at least one non-adjacent location. In some embodiments, the candidate locations may include at least one non-adjacent location and exclude adjacent locations. In some embodiments, the candidate locations may exclude non-adjacent locations and include at least one adjacent location.

[0059] 4 is an illustration of an exemplary implementation of a block and multiple candidate positions according to one embodiment of the present disclosure. Referring to FIGS. 1 and 2, the decoder module 124 may determine multiple adjacent positions 401-405 adjacent to a current block 400 and multiple non-adjacent positions 406-423 not adjacent to the block unit 400. Each of the adjacent positions 401-405 may be adjacent to one of the four corners of the block unit 400. Furthermore, the distance between the block unit 400 and one of the non-adjacent positions 406-423 may be based on one of the block height H, block width W, or diagonal length D of the block unit 400. For example, the horizontal and vertical distances between the top-left position of the block unit 400 and the non-adjacent position 408 may be equal to W+1 and H+1 of the block unit 400, respectively, and the distance between the block unit 400 and the non-adjacent position 411 may be equal to a distance value generated by adding 1 to twice the block height H of the block unit 400.

[0060] In block 340, the decoder module 124 determines a plurality of motion candidates from the plurality of candidate positions.

[0061] 1 and 2, the decoder module 124 may determine multiple neighboring blocks adjacent to the block unit based on the candidate positions. Each of the neighboring blocks may cover at least one of the candidate positions. Each of the candidate positions is located at the upper left position of a corresponding one of the neighboring blocks. Furthermore, the size of the neighboring blocks may be the same as the size of the block unit (i.e., the block height H, the block width W, or the diagonal length D).

[0062] A neighboring block determined based on one of the adjacent locations may cover one or more of the adjacent locations because some of the adjacent locations are adjacent to one another. A neighboring block determined based on one of the non-adjacent locations may cover one of the non-adjacent locations. Furthermore, a neighboring block determined based on one of the non-adjacent locations may cover only one of the non-adjacent locations because the distance between the non-adjacent locations may be equal to or greater than the size of a block unit.

[0063] The decoder module 124 may determine a template region adjacent to the block unit from the image frame. The template region may include at least one of a plurality of adjacent regions. The first adjacent region may be an adjacent region located above the block unit, the second adjacent region may be an adjacent region located to the left of the block unit, and the third adjacent region may be an adjacent region located above and to the left of the block unit. The adjacent regions may be reconstructed before reconstructing the block unit. The height of the first adjacent region may be equal to the number R of reconstructed samples of the first adjacent region along the vertical direction, and the width of the first adjacent region may be equal to the width of the block unit. The height of the second adjacent region may be equal to the height of the block unit, and the width of the second adjacent region may be equal to the number S of reconstructed samples of the second adjacent region along the horizontal direction. Furthermore, the height of the third adjacent region may be equal to the number R of reconstructed samples of the first adjacent region along the vertical direction, and the width of the third adjacent region may be equal to the number S of reconstructed samples of the second adjacent region along the horizontal direction. In one embodiment, the numbers R and S may be positive integers. Furthermore, the numbers R and S may be equal to or different from each other. Furthermore, the numbers R and S may be greater than or equal to 1. In some embodiments, the numbers R and S may be equal to 1.

[0064] The decoder module 124 may determine template regions adjacent to the block unit. The decoder module 124 may use the adjacent regions as template regions for determining motion candidates. Furthermore, the decoder module 124 may use two of the adjacent regions as template regions for determining motion candidates. For example, the decoder module 124 may use only the first and second adjacent regions as template regions for determining motion candidates. Furthermore, the decoder module 124 may use only one of the adjacent regions as a template region for determining motion candidates.

[0065] The decoder module 124 may determine multiple neighboring regions from the image frame. Each of the neighboring regions may be adjacent to a corresponding one of the neighboring blocks. Each of the neighboring regions may include at least one of the multiple neighboring regions of the corresponding one of the neighboring blocks. The first neighboring region of a particular one of the neighboring blocks may be the neighboring region located above the particular neighboring block, the second neighboring region of the particular neighboring block may be the neighboring region located to the left of the particular neighboring block, and the third neighboring region of the particular neighboring block may be the neighboring region located to the upper left of the particular neighboring block. The neighboring regions of the neighboring blocks may be reconstructed before reconstructing the block unit.

[0066] 5A, 5B, and 5C are diagrams illustrating exemplary embodiments of a template region and different neighborhood regions according to an embodiment of the present disclosure. FIG. 5A is a diagram illustrating an exemplary embodiment of a block unit 500, a neighborhood block 530 covering a non-adjacent position 508, and a neighborhood region 5300 according to an embodiment of the present disclosure. The size of the neighborhood block 530 may be the same as the size of the block unit 500. Because the neighborhood block 530 is located above and to the left of the block unit 500 and does not cover the block unit 500, the samples in the neighborhood block 530 may be reconstructed samples reconstructed before the block unit 500. Furthermore, the samples in the neighborhood region 5300 may also be reconstructed samples reconstructed before the block unit 500.

[0067] FIG. 5B is an illustration of an exemplary embodiment of a block unit 500, a neighboring block 540 covering a neighboring position 502, and a neighborhood region 5400, according to an embodiment of the present disclosure. The size of the neighborhood block 540 may be the same as the size of the block unit 500. However, because the neighborhood position 502 is located adjacent to the upper right corner of the block unit 500, some of the samples in the neighborhood block 540 may be reconstructed samples that were reconstructed before the block unit 500, and other samples in the neighborhood block 540 may be unreconstructed samples. Therefore, the decoder module 124 may determine a neighborhood region 5400 that is smaller than the template region 5000 of FIG. 5A to exclude the area covered by the block unit 500. Therefore, because the neighborhood region 5400 is located above the block unit 500 and does not cover the block unit 500, the samples in the neighborhood region 5400 may also be reconstructed samples that were reconstructed before the block unit 500.

[0068] To select a motion candidate for the block unit, the decoder module 124 may determine a neighborhood cost value based on each of the template region and multiple neighboring regions using a cost function. In FIG. 5A , the decoder module 124 may directly compare the neighborhood region 5300 of the neighboring block 530 with the template region 5000 of the block unit 500 using the cost function. For example, the decoder module 124 may compare multiple neighborhood samples in the neighborhood region 5300 of the neighboring block 530 with reconstructed samples in the template region 5000 of the block unit 500 to generate one of the neighborhood cost values. However, because the neighborhood region 5400 may be smaller than the template region 5000, the decoder module 124 may not directly compare the neighborhood region 5400 of the neighboring block 540 with the template region 5000 of the block unit 500 using the cost function. FIG. 5C is an illustration of an exemplary embodiment of a block unit 500 and a secondary region 5001 according to an embodiment of the present disclosure. The secondary region 5001 may be determined based on the size and shape of the neighboring region 5400. The size and shape of the secondary region 5001 may be the same as the size and shape of the neighboring region 5400. Therefore, the decoder module 124 may directly compare the neighboring region 5400 of the neighboring block 540 with the template region 5001 of the block unit 500 using a cost function.

[0069] The cost function may be a template matching cost function, including but not limited to sum of absolute differences (SAD), sum of absolute transformed differences (SATD), mean absolute difference (MAD), mean squared difference (MSD), and structural similarity measure (SSIM). It should be noted that any cost function may be used without departing from this disclosure.

[0070] The decoder module 124 may select multiple candidate blocks based on the neighborhood cost values ​​calculated by the neighborhood regions. However, because the neighborhood regions may be the same or different sizes, directly comparing the neighborhood cost values ​​to each other may be unfair. The decoder module 124 may determine an averaged cost value for each of the neighborhood blocks based on the size of the neighborhood region. For example, the averaged cost value for neighborhood block 530 may be calculated by dividing the neighborhood cost value of neighborhood region 5300 by 8, and the averaged cost value for neighborhood block 540 may be calculated by dividing the neighborhood cost value of neighborhood region 5400 by 5. In this manner, the decoder module 124 may compare the averaged cost values ​​to each other to select some of the neighborhood locations for determining the candidate blocks.

[0071] If the number of selected neighboring positions is equal to 3, three neighboring positions corresponding to the three neighboring blocks with the three lowest averaged cost values ​​may be selected, and the motion candidates may include motion information of three candidate blocks covering the three neighboring positions. If the number of selected neighboring positions is equal to 1, one neighboring position corresponding to the one neighboring block with the lowest averaged cost value may be selected, and the motion candidates may include motion information of one candidate block covering the one neighboring position. Thus, if the number of selected neighboring positions is equal to Y, Y neighboring positions may be selected based on the Y lowest averaged cost values, and Y candidate blocks may be determined. The number Y may be a positive integer.

[0072] The candidate block may be a reconstructed block reconstructed before the block unit is reconstructed. When one of the candidate blocks does not have motion information, the motion candidate may not be determined based on a corresponding selected neighboring position. For example, if one of the candidate blocks is an intra-predicted block, the candidate block does not have motion information to be added to the motion candidate. Furthermore, the motion candidate may further include a history-based candidate determined from a first-in-first-out (FIFO) table that stores multiple previous motion information of multiple previous blocks reconstructed before the block unit is reconstructed.

[0073] At block 350, decoder module 124 selects, based on the motion candidates, multiple collocated blocks each included in one of the one or more collocated frames.

[0074] Each of the plurality of motion candidates may indicate a motion vector, list information, and frame information of a candidate block that covers one of the plurality of candidate positions. Furthermore, the motion vector, list information, and frame information of a particular one of the candidate blocks may be used to predict the particular candidate block in order to reconstruct the particular candidate block.

[0075] The list information for a particular candidate block may include at least one of an inter-prediction indication or a multiple list prediction flag. The inter-prediction indication may indicate whether a first uni-prediction method having a first reference list List0, a second uni-prediction method having a second reference list List1, or a bi-prediction method including the first reference list List0 and the second reference list List1 is used for the particular candidate block. Furthermore, the list prediction flag may include a first list prediction flag used to determine whether the first reference list List0 is used for the particular candidate block and a second list prediction flag used to determine whether the second reference list List1 is used for the particular candidate block. In some embodiments, the list information for a particular candidate block may include both an inter-prediction indication and a multiple list prediction flag for determining the reference list for the particular candidate block. Furthermore, the list information for a particular candidate block may include only an inter-prediction indication for determining the reference list for the particular candidate block. Furthermore, the list information for a particular candidate block may include only a multiple list prediction flag for determining the reference list for the particular candidate block.

[0076] The frame information may include at least one frame index. When a particular candidate block is a bi-predictive block, the frame information of the particular candidate block may include two frame indexes to indicate two reference frames respectively included in the corresponding one of the reference lists. Furthermore, when a particular candidate block is a uni-predictive block, the frame information of the particular candidate block may include only one frame index to indicate one reference frame included in the corresponding one of the reference lists. In another embodiment, when a particular candidate block is a uni-predictive block, the frame information of the particular candidate block may also include two frame indexes. One of the two frame indexes may be used to indicate one reference frame included in the corresponding one of the reference lists, and the other of the two frame indexes may be equal to a predetermined frame value to indicate that the other one of the reference lists is not used for the particular candidate block.

[0077] 1 and 2, the decoder module 124 may determine one or more collocated frames based on at least one predetermined value. If one of the frame indexes in the frame information of a particular candidate block is equal to one of the at least one predetermined value, a reference frame of the particular candidate block selected from a corresponding one of the reference lists indicated by the list information of the particular candidate block may be included in one or more collocated frames. For example, the at least one predetermined value may include zero. In some embodiments, the particular candidate block may be a single predicted block predicted based on the second reference list List1. If the second frame index of the particular candidate block corresponding to the second reference list List1 is equal to zero, a reference frame of the particular candidate block selected from the second reference list List1 based on the second frame index may be included in one or more collocated frames. However, a reference frame of the particular candidate block selected from the second reference list List1 may be excluded from one or more collocated frames if the second frame index in the frame information of the particular candidate block is different from the at least one predetermined value. In this manner, the reference frame of a particular candidate block may be added to or excluded from one or more collocated frames based on the frame information.

[0078] In some embodiments, a particular candidate block may be a bi-predictive block predicted based on a first reference list List0 and a second reference list List1. Furthermore, some of the multiple reference candidates in the first reference list List0 may be identical to some of the multiple reference candidates in the second reference list List1. Thus, a first reference frame of a particular candidate block selected from the first reference list List0 based on a first frame index may be identical to a second reference frame of a particular candidate block selected from the second reference list List1 based on a second frame index. Thus, the number of reference frames of a particular candidate block determined from two reference lists may be equal to one or two when the particular candidate block is a bi-predictive block.

[0079] If the first frame index of a particular candidate block corresponding to the first reference list List0 is equal to 0, the first reference frame of the particular candidate block selected from the first reference list List0 based on the first frame index may be included in one or more collocated frames. Furthermore, if the second frame index of a particular candidate block corresponding to the second reference list List1 is equal to 0, the second reference frame of the particular candidate block selected from the second reference list List1 based on the second frame index may also be included in one or more collocated frames. However, if the frame index in the frame information of the particular candidate block is different from at least one predetermined value, the reference frame of the particular candidate block may be excluded from one or more collocated frames. Therefore, the number of reference frames of a particular candidate block added to one or more collocated frames may be equal to 0, 1, or 2 if the particular candidate block is a bi-predictive block.

[0080] In some embodiments, the decoder module 124 may determine a motion block using a particular one of the motion candidates corresponding to a particular candidate block when the reference frame selected by the frame information of the particular motion candidate is included in one or more collocated frames. The decoder module 124 may select a motion block from a reference frame indicated by the particular motion candidate based on the motion vector of the particular motion candidate. The motion block of the particular motion candidate in the reference frame is indicated by the motion vector of the particular motion candidate from the block unit. In this manner, when one of the frame indexes in the motion candidate is equal to one of the at least one predetermined value, the decoder module 124 may determine a motion block for one of the frame indexes. The number of motion blocks may be equal to the number of frame indexes of the motion candidate equal to one of the at least one predetermined value. Each of the motion blocks can be considered a collocated block. Therefore, each of the collocated blocks may be selected from the reference frame of a corresponding one of the candidate blocks based on a corresponding one of the motion vectors and a corresponding one of the candidate positions.

[0081] In some embodiments, the decoder module 124 may directly use the motion vectors of the motion candidates to determine the motion blocks in each of the one or more collocated frames. The one or more collocated frames may be directly selected from the first reference list List0 and the second reference list List1 based on the at least one predetermined value. The motion block selected by a particular one of the motion candidates is indicated in each of the one or more collocated frames by the motion vector of the particular motion candidate from the block unit. Thus, a particular motion candidate may be used to generate multiple motion blocks, each of which is included in a different collocated frame.

[0082] In some embodiments, because the number of motion blocks generated based on the motion vectors of a particular motion candidate may be greater than one, the decoder module 124 may determine a motion cost value for each of the motion blocks generated in different collocated frames based on the same motion vector. The motion cost value may be determined by a template matching cost function. The decoder module 124 may determine a motion region for each of the motion blocks generated based on the same motion vector and compare the motion region with a block-unit template region. The decoder module 124 may then compare the motion cost values ​​of the motion blocks generated in different collocated frames based on the same motion vector with each other to select only one of the one or more collocated frames that includes one of the motion blocks of the motion vector of the particular motion candidate. In this manner, one of the motion blocks of the particular motion candidate may be set as a collocated block for further comparison with other collocated blocks of other motion candidates. Therefore, the number of collocated blocks may be equal to the number of frame indexes, which may be equal to at least one predetermined value.

[0083] In some embodiments, motion blocks generated in different collocated frames based on the same motion vector may not be compared with each other. Decoder module 124 may generate all motion blocks in one or more collocated frames based on all motion candidates and set all motion blocks as collocated blocks. Thus, the number of motion vectors for determining collocated blocks may be equal to the number of frame indices equal to one of at least one predetermined value, and therefore the number of collocated blocks may be equal to the quantity generated by multiplying the number of frame indices equal to one of at least one predetermined value by the number of one or more collocated frames.

[0084] Returning to FIG. 3, at block 360, the decoder module 124 determines a first cost value for each of the plurality of collocated block units.

[0085] 1 and 2, the decoder module 124 may determine, from one or more collocated frames, multiple collocated regions each adjacent to one of the multiple collocated blocks. Each of the collocated regions may include at least one of the multiple neighboring regions of a corresponding one of the collocated blocks. A first neighboring region of a particular one of the collocated blocks may be an upper neighboring region located above the collocated neighboring block, a second neighboring region of the particular collocated block may be a left neighboring region located to the left of the particular collocated block, and a third neighboring region of the particular collocated block may be an upper-left neighboring region located to the upper left of the particular collocated block. Because each of the one or more collocated frames is reconstructed before the reconstruction of a block unit, the neighboring regions of the collocated blocks may be reconstructed before the reconstruction of the block unit.

[0086] 6 is an illustration of an example embodiment of an image frame 60 having a block unit 600 and a template region 6000, and collocated frames 65-66 having collocated blocks 650-660 and collocated regions 6500-6600, according to one embodiment of the present disclosure. The size of the collocated blocks 650 and 660 may be the same as the size of the block unit 600, and the size of the collocated regions 6500 and 6600 may be the same as the size of the template region 6000. Furthermore, the samples in the collocated blocks 650 and 660 and the collocated regions 6500 and 6600 may also be reconstructed samples that were reconstructed before the block unit 600.

[0087] The decoder module 124 may determine a collocated cost value based on the template region and each of the multiple collocated regions by using a cost function. In Figure 6, the decoder module 124 may directly compare the collocated region 6500 of the collocated block 650 and the collocated region 6600 of the collocated block 660 with the template region 6000 of the block unit 600 by the cost function. For example, the decoder module 124 may compare multiple collocated samples in the collocated region 6500 of the neighboring block 650 with the reconstructed samples in the template region 6000 of the block unit 600 to generate one of the collocated cost values.

[0088] The cost function may be a template matching cost function, including, but not limited to, SAD, SAT, MAD, MSD, and SSIM. It should be noted that any cost function may be used without departing from this disclosure.

[0089] To distinguish between collocated cost values, neighborhood cost values, averaged cost values, and motion cost values, a collocated cost value may be considered a first cost value, a neighborhood cost value may be considered a plurality of second cost values, an averaged cost value may be considered a plurality of third cost values, and a motion cost value may be considered a plurality of fourth cost values. The second cost value and the third cost value may be used only to determine motion candidates in some embodiments of method 300, and therefore the second cost value and the third cost value may not be used in method 300. The fourth cost value may be used to determine collocated blocks in some embodiments of method 300, and therefore the fourth cost value may not be used in method 300. In some embodiments, all of the second cost value, the third cost value, and the fourth cost value may be used in method 300 for determining motion candidates and collocated blocks.

[0090] At block 370, the decoder module 124 determines at least one sub-block-based temporal motion vector prediction (SbTMVP) candidate based on the plurality of motion candidates and the plurality of first cost values.

[0091] The decoder module 124 may select at least one of the collocated blocks based on the first cost value calculated by the collocated region. When the number of selected at least one collocated block is equal to four, four collocated blocks having the four lowest first cost values ​​may be selected, and the decoder module 124 may determine the motion information of the four collocated blocks as four collocated candidates. When the number of selected at least one collocated block is equal to one, the collocated block having the lowest collocated cost value may be selected, and the decoder module 124 may determine the motion information of the selected collocated block as a collocated candidate. Thus, when the number of selected at least one collocated block is equal to Y, Y collocated blocks may be selected based on the Y lowest collocated cost values ​​among the collocated cost values ​​to determine Y collocated candidates. The number Y may be a positive integer. Each of the at least one collocated candidate may indicate motion information, list information, and frame information for a corresponding one of the at least one collocated block.

[0092] Each of the at least one SbTMVP candidates may be determined based on a corresponding one of the at least one collocated candidate. The decoder module 124 may determine a collocated distance between a reference frame of a particular one of the selected at least one collocated block and a particular one of the collocated frames including the particular selected collocated block. The reference frame of the particular selected collocated block may be indicated by motion information (e.g., a collocated candidate) of the particular selected collocated block. Furthermore, the decoder module 124 may determine a reference distance between the reference frame of the block unit and the image frame. Next, motion information of the particular one of the at least one SbTMVP candidate may be determined based on the corresponding one of the collocated distances, the corresponding one of the reference distances, and the motion information of the corresponding one of the collocated candidates. In some embodiments, the reference frame of the block unit may be a corresponding one of the one or more collocated blocks to determine a corresponding one of the at least one SbTMVP candidate.

[0093] Returning to FIG. 3, in block 380, the decoder module 124 reconstructs block units based on a candidate list that includes at least one SbTMVP candidate.

[0094] 1 and 2, decoder module 124 may select a prediction candidate from the candidate list based on a prediction index. The prediction index may indicate a prediction candidate for the block unit from the candidate list. Thus, decoder module 124 may generate a prediction block for the block unit based on the prediction candidate selected from the candidate list. In some embodiments, the candidate list may be a merge list including multiple merge candidates, and at least one SbTMVP candidate may be included in the merge candidate. In some embodiments, the candidate list may be a subblock merge list including multiple subblock merge candidates, and at least one SbTMVP candidate may be included in the subblock merge candidate. In some embodiments, the prediction index may be an index SbTMVP_idx for determining a selected one of the SbTMVP candidates when the candidate list includes only SbTMVP candidates.

[0095] The decoder module 124 may further add multiple residual components to the prediction block to reconstruct the block unit, which may be determined from the bitstream. The decoder module 124 may reconstruct all other block units in the image frame to reconstruct the image frame and video data.

[0096] Another method M1 similar to method 300 may be performed using the configurations shown in FIGS. 1 and 2, the various elements of which are referenced with respect to method 300.

[0097] In method M1, the decoder module 124 may decode the video data based on method 300. Further, in block 370, after selecting at least one of the collocated blocks, the decoder module 124 further determines a plurality of shift blocks each adjacent to one of the selected at least one collocated block. In some embodiments, each of the shift blocks may be selected based on a plurality of neighboring positions, including a plurality of neighboring positions and a plurality of non-neighboring positions in FIG. 4. In some embodiments, each of the shift blocks may be selected based on a plurality of neighboring positions, including four directional positions (e.g., an upper position, a lower position, a left position, and a right position). In some embodiments, each of the shift blocks may be selected based on a plurality of neighboring positions, including eight directional positions (e.g., an upper position, a lower position, a left position, a right position, an upper-left position, an upper-right position, a lower-left position, and a lower-right position).

[0098] 7 is an illustration of an example embodiment of an image frame 70 having a block unit 700 and a template region 7000, and a collocated frame 77 having a collocated block 770, shift blocks 771-774, and collocated regions 7700-7740, according to one embodiment of the present disclosure. The size of the collocated block 770 and the shift blocks 771-774 may be the same as the size of the block unit 700, and the size of the collocated regions 7700-7740 may be the same as the size of the template region 7000. Furthermore, the samples in the collocated block 770, the shift blocks 771-774, and the collocated regions 7700-7740 may also be reconstructed samples that were reconstructed before the block unit 700.

[0099] The decoder module 124 may determine a shift cost value based on the template region and each of the collocated regions by using a cost function. In FIG. 7, the decoder module 124 may directly compare the template region 7000 of the block unit 700 with each of the collocated regions 7700-7740 using the cost function. For example, the decoder module 124 may compare the collocated samples in the collocated region 7740 of the shift block 740 with the reconstructed samples in the template region 7000 of the block unit 700 to generate one of the shift cost values. If one of the shift cost values ​​determined based on the collocated region of the shift block is smaller than the shift cost value determined based on the collocated region of the collocated block, the corresponding one of the shift blocks may replace the collocated block to determine a collocated candidate. In this manner, the collocated candidate may indicate the motion vector, list information, and frame information of the corresponding one of the shift blocks.

[0100] 8 shows a flowchart of a method 800 for decoding video data by an electronic device according to an embodiment of the present disclosure. Method 800 is merely exemplary, as there are various ways to perform the decoding of video data.

[0101] Method 800 may be performed using the configurations illustrated in Figures 1 and 2, where various elements are referenced with respect to method 300. Each block illustrated in Figure 8 may represent one or more processes, methods, or subroutines that are performed.

[0102] The order of the blocks in Figure 8 is exemplary and may be changed, and additional blocks may be added or fewer blocks may be utilized without departing from this disclosure.

[0103] At block 810, the decoder module 124 receives video data. The video data received by the decoder module 124 may be a bitstream.

[0104] 1 and 2, the second electronic device 120 may receive a bitstream from an encoder, such as the first electronic device 110, or another video provider, via a second interface 126. The second interface 126 may provide the bitstream to a decoder module 124.

[0105] The entropy decoding unit 2241 may decode the bitstream to determine prediction instructions and partition instructions for the image frames. The decoder module 124 may then further reconstruct the image frames based on the prediction instructions and partition instructions. The prediction instructions and partition instructions may include flags and indexes.

[0106] In block 820, the decoder module 124 determines block units from the image frame according to the video data.

[0107] 1 and 2, the decoder module 124 may determine an image frame based on a bitstream, and may divide the image frame to determine block units according to a partition instruction in the bitstream. For example, the decoder module 124 may divide the image frame to generate multiple CTUs, and further divide one of the CTUs to determine block units according to a partition instruction based on any video coding standard.

[0108] In block 830, the decoder module 124 determines a number of candidate locations selected from a number of adjacent locations adjacent to the block unit and a number of non-adjacent locations not adjacent to the block unit.

[0109] The candidate locations may be multiple nearby locations adjacent to the block unit. The nearby locations may be selected from at least one of multiple nearby locations adjacent to the block unit or multiple non-adjacent locations not adjacent to the block unit. Thus, in some embodiments, the candidate locations may include at least one of adjacent locations or non-adjacent locations. In some embodiments, the candidate locations may include at least one adjacent location and at least one non-adjacent location. In some embodiments, the candidate locations may include at least one non-adjacent location and exclude adjacent locations. In some embodiments, the candidate locations may exclude non-adjacent locations and include at least one adjacent location.

[0110] 1, 2, and 4, the decoder module 124 may determine a plurality of adjacent positions 401-405 adjacent to the current block 400 and a plurality of non-adjacent positions 406-423 not adjacent to the block unit 400. Each of the adjacent positions 401-405 may be adjacent to one of the four corners of the block unit 400. Furthermore, the distance between the block unit 400 and one of the non-adjacent positions 406-423 may be based on one of the block height H, block width W, or diagonal length D of the block unit 400. For example, the horizontal and vertical distances between the top-left position of the block unit 400 and the non-adjacent position 408 may be equal to W+1 and H+1 of the block unit 400, respectively, and the distance between the block unit 400 and the non-adjacent position 411 may be equal to a distance value generated by adding 1 to twice the block height H of the block unit 400.

[0111] At block 840, the decoder module 124 determines multiple motion candidates from the multiple candidate positions.

[0112] 1 and 2, the decoder module 124 may determine a plurality of neighboring blocks adjacent to the block unit based on the candidate positions. Each of the neighboring blocks may cover at least one of the candidate positions. Each of the candidate positions may be located at an upper left position of a corresponding one of the neighboring blocks. Furthermore, the size of the neighboring blocks may be the same as the size of the block unit (i.e., the block height H, the block width W, or the diagonal length D).

[0113] A neighboring block determined based on one of the adjacent locations may cover one or more of the adjacent locations because some of the adjacent locations are adjacent to one another. A neighboring block determined based on one of the non-adjacent locations may cover one of the non-adjacent locations. Furthermore, a neighboring block determined based on one of the non-adjacent locations may cover only one of the non-adjacent locations because the distance between the non-adjacent locations may be equal to or greater than the size of a block unit.

[0114] The decoder module 124 may determine a template region adjacent to the block unit from the image frame. The template region may include at least one of a plurality of adjacent regions. The first adjacent region may be an adjacent region located above the block unit, the second adjacent region may be an adjacent region located to the left of the block unit, and the third adjacent region may be an adjacent region located above and to the left of the block unit. The adjacent regions may be reconstructed before reconstructing the block unit. The height of the first adjacent region may be equal to the number R of reconstructed samples of the first adjacent region along the vertical direction, and the width of the first adjacent region may be equal to the width of the block unit. The height of the second adjacent region may be equal to the height of the block unit, and the width of the second adjacent region may be equal to the number S of reconstructed samples of the second adjacent region along the horizontal direction. Furthermore, the height of the third adjacent region may be equal to the number R of reconstructed samples of the first adjacent region along the vertical direction, and the width of the third adjacent region may be equal to the number S of reconstructed samples of the second adjacent region along the horizontal direction. In one embodiment, the numbers R and S may be positive integers. Furthermore, the numbers R and S may be equal to or different from each other. Furthermore, the numbers R and S may be greater than or equal to 1. In some embodiments, the numbers R and S may be equal to 1.

[0115] The decoder module 124 may determine template regions adjacent to the block unit. The decoder module 124 may use the adjacent regions as template regions for determining motion candidates. Furthermore, the decoder module 124 may use two of the adjacent regions as template regions for determining motion candidates. For example, the decoder module 124 may use only the first and second adjacent regions as template regions for determining motion candidates. Furthermore, the decoder module 124 may use only one of the adjacent regions as a template region for determining motion candidates.

[0116] The decoder module 124 may determine multiple neighboring regions from the image frame. Each of the neighboring regions may be adjacent to a corresponding one of the neighboring blocks. Each of the neighboring regions may include at least one of the multiple neighboring regions of the corresponding one of the neighboring blocks. The first neighboring region of a particular one of the neighboring blocks may be the neighboring region located above the particular neighboring block, the second neighboring region of the particular neighboring block may be the neighboring region located to the left of the particular neighboring block, and the third neighboring region of the particular neighboring block may be the neighboring region located to the upper left of the particular neighboring block. The neighboring regions of the neighboring blocks may be reconstructed before reconstructing the block unit.

[0117] 5A , the size of the neighboring block 530 may be the same as the size of the block unit 500. Because the neighboring block 530 is located above and to the left of the block unit 500 and does not cover the block unit 500, the samples in the neighboring block 530 may be reconstructed samples that were reconstructed before the block unit 500. Furthermore, the samples in the neighboring region 5300 may also be reconstructed samples that were reconstructed before the block unit 500.

[0118] 5B , the size of the neighborhood block 540 may be the same as the size of the block unit 500. However, because the neighborhood position 502 is located adjacent to the upper right corner of the block unit 500, some of the samples in the neighborhood block 540 may be reconstructed samples that were reconstructed before the block unit 500, and other samples in the neighborhood block 540 may be unreconstructed samples. Therefore, the decoder module 124 may determine a neighborhood region 5400 that is smaller than the template region 5000 in FIG. 5A to exclude the area covered by the block unit 500. Therefore, because the neighborhood region 5400 is located above the block unit 500 and does not cover the block unit 500, the samples in the neighborhood region 5400 may also be reconstructed samples that were reconstructed before the block unit 500.

[0119] To select a motion candidate for the block unit, the decoder module 124 may determine a neighborhood cost value based on the template region and each of multiple neighborhood regions using a cost function. In FIG. 5A, the decoder module 124 may directly compare the neighborhood region 5300 of the neighboring block 530 with the template region 5000 of the block unit 500 using the cost function. For example, the decoder module 124 may compare multiple adjacent samples in the neighborhood region 5300 of the neighboring block 530 with reconstructed samples in the template region 5000 of the block unit 500 to generate one of the neighborhood cost values. However, because the neighborhood region 5400 may be smaller than the template region 5000, the decoder module 124 may not directly compare the neighborhood region 5400 of the neighboring block 540 with the template region 5000 of the block unit 500 using the cost function. Referring to FIG. 5C, a secondary region 5001 may be determined based on the size and shape of the neighborhood region 5400. The size and shape of the secondary region 5001 may be the same as the size and shape of the neighboring region 5400. Therefore, the decoder module 124 may directly compare the neighboring region 5400 of the neighboring block 540 with the template region 5001 of the block unit 500 by a cost function.

[0120] The cost function may be a template matching cost function including, but not limited to, sum of absolute differences (SAD), sum of absolute translation differences (SATD), mean absolute difference (MAD), mean squared difference (MSD), and structural similarity measure (SSIM). It should be noted that any cost function may be used without departing from this disclosure.

[0121] The decoder module 124 may select multiple candidate blocks based on the neighborhood cost values ​​calculated by the neighborhood regions. However, because the neighborhood regions may be the same or different sizes, it may be unfair to directly compare the neighborhood cost values ​​with each other. The decoder module 124 may determine an averaged cost value for each of the neighborhood blocks based on the size of the neighborhood region. For example, the averaged cost value for neighborhood block 530 may be calculated by dividing the neighborhood cost value of neighborhood region 5300 by 8, and the averaged cost value for neighborhood block 540 may be calculated by dividing the neighborhood cost value of neighborhood region 5400 by 5. In this manner, the decoder module 124 may compare the averaged cost values ​​with each other to select some of the neighborhood locations for determining the candidate blocks.

[0122] If the number of selected neighboring positions is equal to 3, three neighboring positions corresponding to the three neighboring blocks with the three lowest averaged cost values ​​may be selected, and the motion candidates may include motion information of three candidate blocks covering the three neighboring positions. If the number of selected neighboring positions is equal to 1, one neighboring position corresponding to the one neighboring block with the lowest averaged cost value may be selected, and the motion candidates may include motion information of one candidate block covering the one neighboring position. Thus, if the number of selected neighboring positions is equal to Y, Y neighboring positions may be selected based on the Y lowest averaged cost values, and Y candidate blocks may be determined. The number Y may be a positive integer.

[0123] The candidate block may be a block reconstructed before reconstructing the block unit. When one of the candidate blocks does not have motion information, the motion candidate may not be determined based on a corresponding selected neighboring position. For example, if one of the candidate blocks is an intra-predicted block, the candidate block does not have motion information to be added to the motion candidate. Furthermore, the motion candidate may further include a history-based candidate determined from a first-in-first-out (FIFO) table that stores multiple previous motion information of multiple previous blocks reconstructed before reconstructing the block unit.

[0124] Returning to FIG. 8, at block 850, decoder module 124 determines, based on the motion candidates, a number of collocated blocks, each of which is included in one of the one or more collocated frames.

[0125] Each of the plurality of motion candidates may indicate a motion vector, list information, and frame information of a candidate block that covers one of the plurality of candidate positions. Furthermore, the motion vector, list information, and frame information of a particular one of the candidate blocks may be used to predict the particular candidate block in order to reconstruct the particular candidate block.

[0126] The list information for a particular candidate block may include at least one of an inter-prediction indication or a multiple list prediction flag. The inter-prediction indication may indicate whether a first uni-prediction method having a first reference list List0, a second uni-prediction method having a second reference list List1, or a bi-prediction method including the first reference list List0 and the second reference list List1 is used for the particular candidate block. Furthermore, the list prediction flag may include a first list prediction flag used to determine whether the first reference list List0 is used for the particular candidate block and a second list prediction flag used to determine whether the second reference list List1 is used for the particular candidate block. In some embodiments, the list information for a particular candidate block may include both an inter-prediction indication and a multiple list prediction flag for determining the reference list for the particular candidate block. Furthermore, the list information for a particular candidate block may include only an inter-prediction indication for determining the reference list for the particular candidate block. Furthermore, the list information for a particular candidate block may include only a multiple list prediction flag for determining the reference list for the particular candidate block.

[0127] The frame information may include at least one frame index. When a particular candidate block is a bi-predictive block, the frame information of the particular candidate block may include two frame indexes to indicate two reference frames respectively included in the corresponding one of the reference lists. Furthermore, when a particular candidate block is a uni-predictive block, the frame information of the particular candidate block may include only one frame index to indicate one reference frame included in the corresponding one of the reference lists. In another embodiment, when a particular candidate block is a uni-predictive block, the frame information of the particular candidate block may also include two frame indexes. One of the two frame indexes may be used to indicate one reference frame included in the corresponding one of the reference lists, and the other of the two frame indexes may be equal to a predetermined frame value to indicate that the other one of the reference lists is not used for the particular candidate block.

[0128] 1 and 2, the decoder module 124 may determine one or more collocated frames based on at least one predetermined value. If one of the frame indexes in the frame information of a particular candidate block is equal to one of the at least one predetermined value, a reference frame of the particular candidate block selected from a reference list indicated by the list information of the particular candidate block may be included in one or more collocated frames. For example, the at least one predetermined value may include zero. In some embodiments, the particular candidate block may be a single predicted block predicted based on the second reference list List1. If the second frame index of the particular candidate block corresponding to the second reference list List1 is equal to zero, a reference frame of the particular candidate block selected from the second reference list List1 based on the second frame index may be included in one or more collocated frames. However, a reference frame of the particular candidate block selected from the second reference list List1 may be excluded from one or more collocated frames if the second frame index in the frame information of the particular candidate block is different from the at least one predetermined value. In this manner, the reference frame of a particular candidate block may be added to or excluded from one or more collocated frames based on the frame information.

[0129] In some embodiments, a particular candidate block may be a bi-predictive block predicted based on a first reference list List0 and a second reference list List1. Furthermore, some of the multiple reference candidates in the first reference list List0 may be identical to some of the multiple reference candidates in the second reference list List1. Thus, a first reference frame of a particular candidate block selected from the first reference list List0 based on a first frame index may be identical to a second reference frame of a particular candidate block selected from the second reference list List1 based on a second frame index. Thus, the number of reference frames of a particular candidate block determined from two reference lists may be equal to one or two when the particular candidate block is a bi-predictive block.

[0130] If the first frame index of a particular candidate block corresponding to the first reference list List0 is equal to 0, the first reference frame of the particular candidate block selected from the first reference list List0 based on the first frame index may be included in one or more collocated frames. Furthermore, if the second frame index of a particular candidate block corresponding to the second reference list List1 is equal to 0, the second reference frame of the particular candidate block selected from the second reference list List1 based on the second frame index may also be included in one or more collocated frames. However, if the frame index in the frame information of the particular candidate block is different from the at least one predetermined value, the reference frame of the particular candidate block may be excluded from one or more collocated frames. Therefore, the number of reference frames of a particular candidate block added to one or more collocated frames may be equal to 0, 1, or 2 if the particular candidate block is a bi-predictive block.

[0131] In some embodiments, the decoder module 124 may determine a motion block using a particular one of the motion candidates corresponding to a particular candidate block when the reference frame selected by the frame information of the particular motion candidate is included in one or more collocated frames. The decoder module 124 may select a motion block from the reference frame indicated by the particular motion candidate based on the motion vector of the particular motion candidate. The motion block of the particular motion candidate in the reference frame is indicated by the motion vector of the particular motion candidate from the block unit. In this manner, when one of the frame indexes in the motion candidate is equal to one of the at least one predetermined value, the decoder module 124 may determine a motion block for one of the frame indexes. The number of motion blocks may be equal to the number of frame indexes of the motion candidate equal to one of the at least one predetermined value. Each of the motion blocks can be considered a collocated block. Therefore, each of the collocated blocks may be selected from the reference frame of the corresponding one of the candidate blocks based on the corresponding one of the motion vectors and the corresponding one of the candidate positions.

[0132] In some embodiments, the decoder module 124 may directly use the motion vectors of the motion candidates to determine the motion blocks in each of the one or more collocated frames. The one or more collocated frames may be directly selected from the first reference list List0 and the second reference list List1 based on the at least one predetermined value. The motion block selected by a particular one of the motion candidates is indicated in each of the one or more collocated frames by the motion vector of the particular motion candidate from the block unit. Thus, a particular motion candidate may be used to generate multiple motion blocks, each of which is included in a different collocated frame.

[0133] In some embodiments, since the number of motion blocks generated based on the motion vector of a particular motion candidate may be greater than one, the decoder module 124 may determine a motion cost value for each of the motion blocks generated in different collocated frames based on the same motion vector. The motion cost value may be determined by a template matching cost function. The decoder module 124 may determine a motion region for each of the motion blocks generated based on the same motion vector and compare the motion region with a block-unit template region. The decoder module 124 may then compare the motion cost values ​​of the motion blocks generated in different collocated frames based on the same motion vector with each other to select only one of the one or more collocated frames that includes one of the motion blocks of the motion vector of the particular motion candidate. In this manner, one of the motion blocks of the particular motion candidate may be set as a collocated block for further comparison with other collocated blocks of other motion candidates. Therefore, the number of collocated blocks may be equal to the number of frame indexes equal to one of the at least one predetermined value.

[0134] In some embodiments, motion blocks generated in different collocated frames based on the same motion vector may not be compared with each other. Decoder module 124 may generate all motion blocks in one or more collocated frames based on all motion candidates and set all motion blocks as collocated blocks. Thus, since the number of motion vectors for determining collocated blocks is equal to the number of frame indices equal to one of the at least one predetermined value, the number of collocated blocks may be equal to the quantity generated by multiplying one of the at least one predetermined value by the number of one or more collocated frames.

[0135] Returning to FIG. 8, in block 860, decoder module 124 derives multiple sub-block-based temporal motion vector prediction (SbTMVP) candidates based on the motion information of multiple collocated blocks.

[0136] 1 and 2, the decoder module 124 may determine whether the collocated block has motion information. If the collocated block is predicted or reconstructed by intra prediction, the collocated block may include only an intra prediction mode that does not include motion information. If the collocated block is predicted or reconstructed by inter prediction, the decoder module 124 may receive motion information of the collocated block.

[0137] The motion information for each of the collocated blocks may include vector information, list information, and frame information for a corresponding one of the collocated blocks. Furthermore, the vector information, list information, and frame information for a particular one of the collocated blocks may be used to predict the particular collocated block in order to reconstruct the particular collocated block.

[0138] The decoder module 124 may use the motion information of the collocated blocks to determine SbTMVP candidates, each of which corresponds to one of the collocated blocks. In some embodiments, the decoder module 124 may determine an initial SbTMVP list including all of the SbTMVP candidates for the block unit. When a block unit is predicted based on the SbTMVP candidates, the block unit may be divided into multiple subblock units. Accordingly, each of the collocated blocks may also be divided into multiple collocated subblocks. Before reconstructing the image frame, the decoder module 124 may use the motion information of the collocated blocks to determine multiple first subblock vectors Vs1 for the collocated subblocks as the collocated frame is reconstructed. Next, the decoder module 124 may derive multiple second subblock vectors Vs2 for the subblock unit based on the multiple first subblock vectors Vs1. For example, decoder module 124 may use motion information of a particular collocated block to determine M first sub-block vectors Vs1 for M collocated sub-blocks within the particular collocated block. Then, decoder module 124 may derive M second sub-block vectors Vs2 for M sub-block units within the block unit based on the M first sub-block vectors Vs1. Each of the M second sub-block vectors Vs2 may be derived based on a corresponding one of the M first sub-block vectors Vs1, a reference distance between the image frame and the reference frame of the block unit, and a collocated distance between the collocated frame and the collocated reference frame of the collocated block.

[0139] Furthermore, when a particular one of the collocated subblocks is included in a uni-predictive block, decoder module 124 may derive one second subblock vector for a particular one of the subblock units corresponding to the particular collocated subblock. Furthermore, when a particular collocated subblock is included in a bi-predictive block, decoder module 124 may derive two second subblock vectors for a particular subblock unit corresponding to the particular collocated subblock. In some embodiments, some of the collocated subblocks in a particular collocated block may be uni-predictive blocks, and other of the collocated subblocks in a particular collocated block may be bi-predictive blocks. Thus, when decoder module 124 determines the M collocated subblocks in a particular collocated block, decoder module 124 may use motion information of the M collocated subblocks to determine (M+X) first subblock vectors Vs1 for the M collocated subblocks in the particular collocated block. Next, decoder module 124 may derive (M+X) second sub-block vectors Vs2 for the M sub-block units in the block unit based on the (M+X) first sub-block vectors Vs1. The number X may be equal to the number of bi-predictive collocated sub-blocks in a particular collocated block. The number X may be in the range of 0 to M. Thus, the number of second sub-block vectors Vs2 may be in the range of M to 2M.

[0140] Because the M first subblock vectors Vs1 of the collocated subblocks received from the differential encoding unit may differ from one another, the M second subblock vectors Vs2 of the subblock unit may differ from one another. Thus, when decoder module 124 uses the M second subblock vectors Vs2 determined from a particular one of the collocated blocks to determine one of the SbTMVP candidates, decoder module 124 may determine multiple reference subblocks that are not adjacent to one another because the M second subblock vectors Vs2 corresponding to the particular collocated block may differ from one another. In some embodiments, each of the SbTMVP candidates may indicate the M first subblock vectors Vs1 for a corresponding one of the collocated blocks. In some embodiments, each of the SbTMVP candidates may indicate the M second subblock vectors Vs2 corresponding to one of the collocated blocks for the block unit. In some embodiments, each of the SbTMVP candidates may indicate a reference sub-block corresponding to one of the collocated blocks for the block unit.

[0141] Returning to FIG. 8, at block 870, the decoder module 124 determines a first cost value based on each of the plurality of SbTMVP candidates.

[0142] 1 and 2, the decoder module 124 may determine template regions adjacent to the block unit from the image frame. The template region may include at least one of a plurality of adjacent regions. The first adjacent region may be an adjacent region located above the block unit, the second adjacent region may be an adjacent region located to the left of the block unit, and the third adjacent region may be an adjacent region located above and to the left of the block unit. The adjacent regions may be reconstructed before reconstructing the block unit.

[0143] The decoder module 124 may further determine a reference region including multiple reference sub-regions generated for reference sub-blocks corresponding to one of the collocated blocks. Because the reference sub-blocks corresponding to a particular one of the collocated blocks may not be adjacent to each other, the reference sub-region corresponding to the particular collocated block may be determined based on some of the M second sub-block vectors Vs2. Some of the M second sub-block vectors Vs2 may belong to a sub-block unit located at the top boundary of the block unit or may belong to a sub-block unit located at the left boundary of the block unit.

[0144] 9A and 9B are diagrams illustrating exemplary implementations of a template region and a reference region according to an embodiment of the present disclosure. Fig. 9A is a diagram illustrating an exemplary implementation of a block unit 900 and a template region 900 according to an embodiment of the present disclosure. A template region 9000 adjacent to the block unit 900 is determined in an image frame. The template region 9000 may be reconstructed before reconstructing the block unit 900.

[0145] 9B is an illustration of an exemplary embodiment of multiple reference sub-blocks 901-916 and a reference region according to an embodiment of the present disclosure. The reference region may include multiple reference sub-regions 9011, 9051, 9091, 9131, 9132, 9142, 9152, and 9162. The reference sub-regions 9011, 9051, 9091, and 9131 may be located to the left of the reference sub-blocks 901, 905, 909, and 913, respectively. The reference sub-regions 9132, 9142, 9152, and 9162 may be located above the reference sub-blocks 913-916, respectively. The reference sub-blocks 901, 905, 909, and 913 may be determined based on the sub-block unit located at the left boundary of the block unit 900 and the second sub-block vector Vs2 of the sub-block unit located at the left boundary. The reference sub-blocks 913 to 916 may be determined based on the sub-block unit located at the upper boundary of the block unit 900 and the second sub-block vector Vs2 of the sub-block unit located at the upper boundary.

[0146] The decoder module 124 may determine a reference cost value based on the template region and each of the multiple reference regions using a cost function. The decoder module 124 may directly compare the template region 9000 of the block unit 900 with the reference regions that respectively correspond to one of the collocated blocks using the cost function. For example, the decoder module 124 may compare the reconstructed samples in a particular one of the reference regions that corresponds to a particular one of the collocated blocks with the reconstructed samples in the template region 9000 of the block unit 900 to generate one of the reference cost values.

[0147] The cost function may be a template matching cost function, including, but not limited to, SAD, SAT, MAD, MSD, and SSIM. It should be noted that any cost function may be used without departing from this disclosure.

[0148] To distinguish the reference cost value, the neighborhood cost value, the averaged cost value, and the motion cost value from one another, the reference cost value may be a first cost value, the neighborhood cost values ​​may be a plurality of second cost values, the averaged cost values ​​may be a plurality of third cost values, and the motion cost values ​​may be a plurality of fourth cost values. Because the second cost value and the third cost value may be used only to determine motion candidates in some embodiments of method 800, it may be unnecessary to use the second cost value and the third cost value in method 800. Because the fourth cost value may be used to determine collocated blocks in some embodiments of method 800, it may be unnecessary to use the fourth cost value in method 800. In some embodiments, all of the second cost value, the third cost value, and the fourth cost value may be used in method 800 for determining motion candidates and collocated blocks.

[0149] In block 880, the decoder module 124 determines an arrangement of the multiple SbTMVP candidates based on the first cost values ​​and generates an adjusted SbTMVP list based on the arrangement.

[0150] 1 and 2, decoder module 124 may determine the arrangement of the SbTMVP candidates based on the first cost value and sort the SbTMVP candidates based on the arrangement. In some embodiments, the SbTMVP candidates may be sorted in ascending or descending order of the first cost value.

[0151] Before the placement is determined based on the first cost value, the SbTMVP candidates may be ordered based on any rule, such as ordering the SbTMVP candidates by adjacent and non-adjacent positions.

[0152] SbTMVP candidates may be sorted in ascending order of reference cost value. Thus, if the reference cost value of one particular SbTMVP candidate is lower than the reference cost values ​​of the other SbTMVP candidates, the particular SbTMVP candidate may be advanced to become the first SbTMVP candidate based on the arrangement. In other words, if the reference cost value of a particular SbTMVP candidate is the smallest of the reference cost values ​​of the SbTMVP candidates, the particular SbTMVP candidate may be promoted to the first SbTMVP candidate. Furthermore, if the reference cost value of a particular SbTMVP candidate is the largest of the reference cost values ​​of the SbTMVP candidates, the particular SbTMVP candidate may be moved to the last of the SbTMVP candidates. For example, if the number of SbTMVP candidates is six, the SbTMVP candidates may include six SbTMVP candidates SC1, SC2, SC3, SC4, SC5, and SC6, each with six reference cost values ​​(CV1, CV2, CV3, CV4, CV5, and CV6). Furthermore, if the order of the six reference cost values ​​is CV4>CV2>CV5>CV1>CV6>CV3, the arrangement of the SbTMVP candidates may be changed from the original order of the six SbTMVP candidates SC1, SC2, SC3, SC4, SC5, and SC6 to a new order of the six SbTMVP candidates SC3, SC6, SC1, SC5, SC2, and SC4.

[0153] The arrangement may further be determined by adjusting the new order of the SbTMVP candidates based on the diversity reference. The decoder module 124 may determine the difference between two of the reference cost values ​​selected from two adjacent ones of the order of the SbTMVP candidates in the new order. If the difference value is equal to or less than the diversity threshold, the last of the two adjacent SbTMVP candidates may be moved back. For example, the order of the six reference cost values ​​is CV4 > CV2 > CV5 > CV1 > CV6 > CV3, and the difference between the two reference cost values ​​CV1 and CV6 is equal to or less than the diversity threshold. Therefore, SbTMVP candidate SC1 may be moved back. Therefore, the arrangement of the SbTMVP candidates may be further changed from the new order of the six SbTMVP candidates SC3, SC6, SC1, SC5, SC2, SC4 to the final order of the six SbTMVP candidates SC3, SC6, SC5, SC1, SC2, SC4. In some embodiments, the placement may be identical to the new order of the SbTMVP candidates, as the placement may not be determined by adjusting the new order of the SbTMVP candidates based on a diversity criterion.

[0154] Decoder module 124 may select K SbTMVP candidates with the smallest reference cost values ​​from the SbTMVP candidates and add the selected SbTMVP candidates to the adjusted SbTMVP list. The positive integer number K may be equal to the number of SbTMVP candidates in the adjusted SbTMVP list and may be less than the total number of SbTMVP candidates. In other words, when decoder module 124 sorts the SbTMVP candidates in ascending order of reference cost value to generate an arrangement, it may select the first through K SbTMVP candidates sorted based on that arrangement. In some embodiments, the adjusted SbTMVP list may differ from the initial SbTMVP list if the order of the initial cost values ​​differs from the original order of the SbTMVP candidates in the initial SbTMVP list. In some embodiments, the adjusted SbTMVP list may differ from the initial SbTMVP list if the number of the K selected SbTMVP candidates differs from the number of SbTMVP candidates in the initial SbTMVP list. In some embodiments, the adjusted SbTMVP list may be identical to the initial SbTMVP list if the order of the initial cost values ​​is the same as the original order of the SbTMVP candidates in the initial SbTMVP list and the number of K selected SbTMVP candidates is equal to the number of SbTMVP candidates in the initial SbTMVP list.

[0155] Each SbTMVP candidate in the adjusted SbTMVP list may have an SbTMVP index. Therefore, since the number of SbTMVP candidates in the adjusted SbTMVP list is equal to K, the SbTMVP index of the adjusted SbTMVP list may be in the index range of 0 to K-1.

[0156] SbTMVP candidates can be divided into multiple candidate groups based on multiple division criteria. For example, SbTMVP candidates can be divided into multiple candidate groups based on neighborhood position type, block unit block size, or at least one predetermined number. When dividing SbTMVP candidates based on neighborhood position type, SbTMVP candidates generated based on non-adjacent positions can be included in one candidate group, SbTMVP candidates generated based on adjacent positions can be included in another candidate group, and SbTMVP candidates with a sub-block motion vector of zero can be included in another candidate group. For example, the number of SbTMVP candidates can be equal to 10. There can be two SbTMVP candidates generated based on adjacent positions, seven SbTMVP candidates generated based on non-adjacent positions, and one SbTMVP candidate with a sub-block motion vector of zero. Therefore, SbTMVP candidates can be divided into three candidate groups: a first candidate group with two SbTMVP candidates, a second candidate group with seven SbTMVP candidates, and a third candidate group with one SbTMVP candidate.

[0157] Furthermore, if the SbTMVP candidates are divided based on the block size of the block unit, decoder module 124 may determine the group size of the candidate group based on the block size of the block unit. For example, if the block size is 8x8, the group size may be equal to 3, and if the block size is 16x16, the group size may be equal to 5. In this manner, decoder module 124 may determine the group size based on the block size from the lookup table. Furthermore, the group sizes of the candidate groups may be equal to each other. In some embodiments, the block size may be the width of the block unit. In some embodiments, the block size may be the height of the block unit. In some embodiments, the block size may be a size value generated by multiplying the width of the block unit by the height of the block unit. In some embodiments, the block size may be a size value generated by dividing the width of the block unit by the height of the block unit. In some embodiments, the block size may be a size value generated by dividing the height of the block unit by the width of the block unit. In some embodiments, the block size may be calculated based on the width of the block unit and / or the height of the block unit.

[0158] If the SbTMVP candidates are divided based on the at least one predetermined number, decoder module 124 may determine the group size of the candidate groups based on the at least one predetermined number. If the group sizes of the candidate groups are equal to one another, the at least one predetermined number may be equal to 1. If some of the group sizes of the candidate groups are different from one another, the at least one predetermined number may be greater than 1. Thus, decoder module 124 may divide the SbTMVP candidates into multiple candidate groups, each having the same or different group sizes. Furthermore, if the at least one predetermined number is equal to 1, the group size of the last candidate group may be different from the group sizes of the other candidate groups because the number of SbTMVP candidates may not be divisible by the predetermined number. In some embodiments, the number of SbTMVP candidates may be equal to 5. The predetermined number for dividing the SbTMVP candidates may be equal to 3. The SbTMVP candidates in the first candidate group may be candidates SbTMVP0, SbTMVP1, and SbTMVP2 with corresponding indices 0, 1, and 2, respectively, and the SbTMVP candidates in the second candidate group may be candidates SbTMVP3 and SbTMVP4 with corresponding indices 3 and 4, respectively.

[0159] In some embodiments, the group sizes of the candidate groups may differ from one another. Thus, if the number of candidate groups is equal to x, the group sizes may be equal to N1, N2, N3, ..., Nx. For example, SbTMVP candidates may be divided into two candidate groups. The first candidate group may include the first N1 SbTMVP candidates with indices ranging from 0 to (N1-1), and the second candidate group may include the N2 SbTMVP candidates with indices ranging from N1 to (N1+N2-1). In some embodiments, the number of SbTMVP candidates may be equal to 15. The predetermined number for dividing the SbTMVP candidates may be equal to 4, 5, and 6. Thus, the SbTMVP candidates may be divided into three candidate groups, each with a different group size. The first subgroup may include four SbTMVP candidates with indices ranging from 0 to 3, and the second subgroup may include five SbTMVP candidates with indices ranging from 4 to 8. The third subgroup may include six SbTMVP candidates with indices from 9 to 14.

[0160] SbTMVP candidates may be collected based on a collection index used by decoder module 124 to determine how many SbTMVP candidates to collect in order to select one of the SbTMVP candidates. For example, if the collection index indicates that the selected SbTMVP candidate is included in the pth candidate group, the number of SbTMVP candidates collected by decoder module 124 may be greater than the sum of N1, N2, ..., Np. In other words, decoder module 124 may collect only P SbTMVP candidates, and the amount of P SbTMVP candidates may be sufficient to distribute to the first pth candidate group. In some embodiments, the collection index may be a candidate group index indicating which candidate group includes the selected SbTMVP candidate. In some embodiments, a prediction index that directly indicates the selected SbTMVP candidate may be used as the collection index to derive the number of collected SbTMVP candidates. For example, the number of SbTMVP candidates may be equal to 15, and the predetermined number for dividing the SbTMVP candidates may be equal to 4, 5, or 6. The selected SbTMVP candidate may be included in the second candidate group if the analyzed prediction index is equal to 5. Thus, decoder module 124 may stop collecting SbTMVP candidates once nine SbTMVP candidates have been collected.

[0161] In some embodiments, the SbTMVP candidates in each candidate group may be sorted in ascending order of the reference cost values ​​of the SbTMVP candidates in the corresponding one of the candidate groups. Thus, if the reference cost value of a particular SbTMVP candidate in a particular one of the candidate groups is lower than the reference cost values ​​of the other SbTMVP candidates in the particular candidate group, the particular SbTMVP candidate may be advanced to become the first SbTMVP candidate in the particular candidate group based on the arrangement. In other words, if the reference cost value of a particular SbTMVP candidate in a particular candidate group is the smallest value of the reference cost values ​​for the particular candidate group, the particular SbTMVP candidate may be advanced to become the first SbTMVP candidate in the particular candidate group.

[0162] Returning to FIG. 8, in block 890, the decoder module 124 reconstructs the block units based on the adjusted SbTMVP list.

[0163] 1 and 2, decoder module 124 may select one of the SbTMVP candidates ordered based on the arrangement from the adjusted SbTMVP list based on a prediction index. The prediction index may indicate a prediction candidate for a block unit from the SbTMVP candidates ordered based on the arrangement of the adjusted SbTMVP list. In some embodiments, the prediction index may be an index SbTMVP_idx for determining the selected one of the SbTMVP candidates.

[0164] The index value of the SbTMVP index in the adjusted SbTMVP list may be within the index range of 0 to K-1. Therefore, since the index value of the SbTMVP index may not be greater than K-1, the SbTMVP candidates arranged by array after the Kth candidate may be excluded from the selection of predicted candidates.

[0165] In some embodiments, SbTMVP candidates sorted based on their placement in the adjusted SbTMVP list may be added to a candidate list. Decoder module 124 may select one of a plurality of mode candidates including the SbTMVP candidates sorted based on their placement based on a prediction index. The prediction index may indicate a prediction candidate for a block unit. For example, the candidate list may be a subblock merge list, and the prediction index may be a subblock merge index. Furthermore, the candidate list may be a merge list, and the prediction index may be a merge index.

[0166] Decoder module 124 may determine a selected one of the SbTMVP candidates based on an index in the bitstream, determine a prediction candidate based on the selected SbTMVP candidate, and then generate a prediction block for the block unit based on the selected SbTMVP candidate.

[0167] The decoder module 124 may further add multiple residual components to the prediction block to reconstruct the block unit. The residual components may be determined from the bitstream. The decoder module 124 may reconstruct all other block units in the image frame to reconstruct the image frame and video data.

[0168] Another method M2, generated by combining methods 300 and 800, may be performed using the configurations shown in Figures 1 and 2, the various elements of which are referenced with respect to methods 300 and 800.

[0169] In method M2, decoder module 124 may decode video data based on blocks 310-360 of method 300. Then, decoder module 124 may select a portion of collocated blocks based on the collocated cost values ​​of the collocated blocks. For example, when the number of the portion of collocated blocks is equal to Y, Y collocated blocks may be selected based on the Y lowest of the collocated cost values ​​to determine Y collocated candidates. The number Y may be a positive integer.

[0170] In method M2, after the Y collocated blocks are selected, decoder module 124 may decode the video data based on blocks 860 to 890 of method 800. In this manner, decoder module 124 may derive SbTMVP candidates based on the motion information of the Y collocated blocks for reconstructing the block unit.

[0171] 10 shows a block diagram of the encoder module 114 of the first electronic device 110 shown in FIG. 1 according to an embodiment of the present disclosure. The encoder module 114 may include a prediction processor (e.g., a prediction process unit 10141), at least a first adder (e.g., a first adder 10142) and a second adder (e.g., a second adder 10145), a transform / quantization processor (e.g., a transform / quantization unit 10143), an inverse quantization / inverse transform processor (e.g., an inverse quantization / inverse transform unit 10144), a filter (e.g., a filtering unit 10146), a decoded picture buffer (e.g., a decoded picture buffer 10147), and an entropy encoder (e.g., an entropy encoding unit 10148). The prediction process unit 10141 of the encoder module 114 may further include a partition processor (e.g., partition unit 101411), an intra-prediction processor (e.g., intra-prediction unit 101412), and an inter-prediction processor (e.g., inter-prediction unit 101413).

[0172] The encoder module 114 may receive a source video, encode the source video, and output a bitstream. The encoder module 114 may receive a source video including a plurality of image frames, and divide the image frames according to a coding structure. Each image frame may be divided into at least one image block.

[0173] The at least one image block may include a luminance block having a plurality of luminance samples and at least one chrominance block having a plurality of chrominance samples, and the luminance block and the at least one chrominance block may be further divided to generate a macroblock, a coding tree unit (CTU), a coding block (CB), a subdivision thereof, and / or another equivalent coding unit.

[0174] Encoder module 114 may perform additional subdivisions of the source video. Note that the disclosed embodiments are applicable to video coding generally, regardless of how the source video is divided before and / or during coding.

[0175] During the coding process, the prediction process unit 10141 may receive a current image block of an image frame, which may be a luminance block or a chrominance block of the image frame.

[0176] The partition unit 101411 may divide the current image block into multiple block units. The intra prediction unit 101412 may perform intra predictive coding of the current block unit relative to one or more neighboring blocks in the same frame as the current block unit to provide spatial prediction. The inter prediction unit 101413 may perform inter predictive coding of the current block unit relative to one or more blocks in one or more reference image blocks to provide temporal prediction.

[0177] The prediction process unit 10141 may select one of the loading results generated by the intra prediction unit 101412 and the inter prediction unit 101413 based on a mode selection method such as a cost function. The mode selection method may be a rate-distortion optimization (RDO) process.

[0178] The prediction process unit 10141 may determine a selected coding result and provide a predicted block corresponding to the selected coding result to a first adder 10142 for generating a residual block and a second adder 10145 for reconstructing an encoded block unit. The prediction process unit 10141 may further provide syntax elements such as motion vectors, intra-mode indicators, partition information, and other syntax information to the entropy encoding unit 10148.

[0179] The intra prediction unit 101412 may intra predict the current block unit. The intra prediction unit 101412 may determine an intra prediction mode directed to reconstructed samples neighboring the current block unit to encode the current block unit.

[0180] The intra prediction unit 101412 may encode the current block unit using various intra prediction modes. The intra prediction unit 101412 of the prediction process unit 10141 may select an appropriate intra prediction mode from the selected modes. The intra prediction unit 101412 may encode the current block unit using a cross-component prediction mode and predict one of the two chroma components of the current block unit based on the luma component of the current block unit. The intra prediction unit 101412 may predict a first of the two chroma components of the current block unit based on a second of the two chroma components of the current block unit.

[0181] The inter prediction unit 101413 may inter predict the current block unit as an alternative to the intra prediction performed by the intra prediction unit 101412. The inter prediction unit 101413 may perform motion estimation to estimate the motion of the current block unit to generate a motion vector.

[0182] The motion vector may indicate the displacement of a current block unit in the current image block relative to a reference block unit in the reference image block. The inter prediction unit 101413 may receive at least one reference image block stored in the decoded picture buffer 10147 and estimate motion based on the received reference image block to generate a motion vector.

[0183] The first adder 10142 may generate a residual block by subtracting the predicted block determined by the prediction process unit 10141 from the original current block unit. The first adder 10142 may represent a component that performs this subtraction.

[0184] The transform / quantization unit 10143 may apply a transform to the residual blocks to generate residual transform coefficients and may quantize the residual transform coefficients to further reduce the bit rate. The transform may be one of a DCT, DST, AMT, MDNSST, HyGT, signal-dependent transform, KLT, wavelet transform, integer transform, sub-band transform, or conceptually similar transform.

[0185] The transform may convert the residual information from the pixel value domain to a transform domain, such as the frequency domain. The degree of quantization can be varied by adjusting a quantization parameter.

[0186] The transform / quantization unit 10143 may perform the scan of the matrix containing the quantized transform coefficients, or the entropy encoding unit 10148 may perform the scan.

[0187] The entropy encoding unit 10148 may receive multiple syntax elements, including quantization parameters, transform data, motion vectors, intra modes, partition information, and other syntax information, from the prediction processing unit 10141 and the transform / quantization unit 10143. The entropy encoding unit 10148 may encode the syntax elements into a bitstream.

[0188] The entropy encoding unit 10148 may entropy encode the quantized transform coefficients by performing CAVLC, CABAC, SBAC, PIPE coding, or another entropy coding technique to generate an encoded bitstream that may be transmitted to another device (i.e., the second electronic device 120 of FIG. 1) or archived for later transmission or retrieval.

[0189] The inverse quantization / inverse transform unit 10144 may apply inverse quantization and inverse transform to reconstruct the residual block in the pixel domain for later use as a reference block. The second adder 10145 may add the reconstructed residual block to the prediction block provided from the prediction process unit 10141 to generate a reconstructed block for storage in the decoded picture buffer 10147.

[0190] The filtering unit 10146 may include a deblocking filter, an SAO filter, a bilateral filter, and / or an ALF to remove blocking artifacts from the reconstructed blocks. In addition to the deblocking filter, the SAO filter, the bilateral filter, and the ALF, additional filters (in-loop or post-loop) may be used. Such filters are not shown for simplicity and may filter the output of the second adder 10145.

[0191] The decoded picture buffer 10147 may be a reference picture memory that stores reference blocks used by the encoder module 114 to encode video, such as in intra-coding or inter-coding modes. The decoded picture buffer 10147 may include various memory devices, such as DRAM (including, for example, SDRAM), MRAM, RRAM, or other types of memory devices. The decoded picture buffer 10147 may be on-chip with other components of the encoder module 114 or off-chip relative to those components.

[0192] The method 300 for decoding / encoding video data may be performed by the first electronic device 110. The encoder module 114 may receive video data. The video data received by the encoder module 114 may be a video. The encoder module 114 may determine block units from an image according to the video data. The encoder module 114 may divide an image frame to generate a plurality of CTUs, and further divide one of the CTUs to determine the block units according to one of a plurality of partition schemes based on any video coding standard.

[0193] 1, 3, and 10, the encoder module 114 may determine a plurality of candidate positions, including at least one non-adjacent position that is not adjacent to the block unit. The candidate positions may be a plurality of neighboring positions that are adjacent to the block unit. The neighboring positions may be selected from at least one of a plurality of neighboring positions that are adjacent to the block unit or a plurality of non-adjacent positions that are not adjacent to the block unit.

[0194] Referring to FIGS. 1, 3, and 10, the encoder module 114 may determine multiple motion candidates from multiple candidate positions. The encoder module 114 may determine multiple neighboring blocks adjacent to the block unit based on the candidate positions. Each of the neighboring blocks may cover at least one of the candidate positions. The encoder module 114 may determine a template region adjacent to the block unit from the image frame and determine multiple neighboring regions from the image frame. Each of the neighboring regions may be adjacent to a corresponding one of the neighboring blocks. The encoder module 114 may determine a neighborhood cost value based on the template region and each of the multiple neighboring regions using a cost function for selecting a motion candidate for the block unit. The encoder module 114 may select multiple candidate blocks based on the neighborhood cost values ​​calculated by the neighboring regions. The candidate blocks may be reconstructed blocks reconstructed before reconstructing the block unit. If one of the candidate blocks does not have motion information, a motion candidate may not be determined based on the corresponding one of the selected neighboring positions. Additionally, the motion candidates may further include history-based candidates determined from a first-in-first-out (FIFO) table that stores multiple previous motion information of multiple previous blocks that were reconstructed before reconstructing the block unit.

[0195] 1, 3, and 10, the encoder module 114 may select multiple collocated blocks, each included in one of one or more collocated frames, based on multiple motion candidates. Each of the multiple motion candidates may indicate a motion vector, list information, and frame information of a candidate block covering one of the multiple candidate positions. The encoder module 114 may determine one or more collocated frames based on at least one predetermined value. When one of multiple frame indexes in the frame information of a specific one of the candidate blocks is equal to one of the at least one predetermined value, a reference frame of the specific candidate block selected from a corresponding one of the reference lists indicated by the list information of the specific candidate block may be included in one or more collocated frames. The encoder module 114 may determine a motion block using a specific one of the motion candidates corresponding to a specific candidate block if the reference frame selected by the frame information of the specific motion candidate is included in one or more collocated frames. Each motion block may be considered a collocated block. In some embodiments, the encoder module 114 may determine a motion cost value for each of the motion blocks generated in different collocated frames based on the same motion vector, and the encoder module 114 may compare the motion cost values ​​of the motion blocks to select a collocated block.

[0196] 1, 3, and 10, the encoder module 114 may determine a collocated cost value for each of a plurality of collocated block units. The encoder module 114 may determine a plurality of collocated regions from one or more collocated frames, each of which is adjacent to one of the plurality of collocated blocks. Each of the collocated regions may include at least one of a plurality of adjacent regions of a corresponding one of the collocated blocks. The encoder module 114 may determine a collocated cost value based on the template region and each of the plurality of collocated regions using a cost function.

[0197] 1, 3, and 10, the encoder module 114 determines at least one sub-block-based temporal motion vector prediction (SbTMVP) candidate based on a plurality of motion candidates and a plurality of first cost values. The encoder module 114 may select at least one of the collocated blocks based on the first cost value calculated by the collocated region. Each of the at least one SbTMVP candidate may be determined based on a corresponding one of the at least one collocated candidate.

[0198] 1, 3, and 10, the encoder module 114 may select one of a plurality of mode candidates in a candidate list including at least one SbTMVP candidate based on a mode selection method, such as a cost function for predicting the block unit. The encoder module 114 may predict the block unit based on at least one SbTMVP candidate in the candidate list to generate a plurality of predicted blocks. Next, the encoder module 114 may further compare a plurality of predicted samples included in each of the predicted blocks with a plurality of color components in the block unit. In this manner, the encoder module 114 may determine a block cost value for each predicted block and select a prediction candidate for the block unit from the mode candidates in the candidate list based on the block cost value. The mode selection method may be a rate-distortion optimization (RDO) process. Next, the encoder module 114 may determine a plurality of residual components by comparing the color components of the block unit with the predicted samples generated based on the prediction mode. Furthermore, the residual components and a prediction index may be encoded into a bitstream for a decoder device to reconstruct video data. The prediction index may indicate a selected one of the mode candidates in the candidate list. In some embodiments, the prediction index may be an index SbTMVP_idx for determining a selected one of the SbTMVP candidates when the candidate list includes only SbTMVP candidates.

[0199] 1, 3, and 10, the encoder module 114 may further reconstruct the block unit based on the prediction candidates in the candidate list to generate a reconstructed block including a plurality of reconstructed samples, which may be used as references for predicting a plurality of subsequent blocks in the video data.

[0200] Therefore, the encoder module 114 may also use the method 300 to predict and reconstruct block units for encoding video data into a bitstream. Because the encoder module 114 and the decoder module 124 may use the same method 300, the encoder module 114 and the decoder module 124 may generate the same reordering list to reduce the number of bits in the bitstream.

[0201] A method M1 similar to the method 300 for decoding / encoding video data may be performed by the first electronic device 110. In method M1, the encoder module 114 may encode the video data based on method 300. Further, in block 370, after selecting at least one of the collocated blocks, the encoder module 114 further determines each of a plurality of shift blocks adjacent to one of the selected at least one collocated block. In some embodiments, each of the shift blocks may be selected based on a plurality of neighboring positions, including a plurality of adjacent positions and a plurality of non-adjacent positions in FIG. 4 .

[0202] 1, 3, and 10, the encoder module 114 may determine a shift cost value based on the template region and each of the plurality of collocated regions by using a cost function. In FIG. 7, the encoder module 114 may directly compare the template region 7000 of the block unit 700 with each of the collocated regions 7700-7740 by the cost function. If one of the shift cost values ​​determined based on the collocated region of the shift block is smaller than the shift cost value determined based on the collocated region of the collocated block, the corresponding one of the shift blocks may replace the collocated block for determining a collocated candidate.

[0203] The method 800 for decoding / encoding video data may be performed by the first electronic device 110. The encoder module 114 may receive video data. The video data received by the encoder module 114 may be a video. The encoder module 114 may determine block units from an image according to the video data. The encoder module 114 may divide an image frame to generate a plurality of CTUs, and further divide one of the CTUs to determine the block units according to one of a plurality of division schemes based on any video coding standard.

[0204] 1, 8, and 10, the encoder module 114 may determine a plurality of candidate locations selected from a plurality of adjacent locations adjacent to the block unit and a plurality of non-adjacent locations not adjacent to the block unit. The candidate locations may be a plurality of nearby locations adjacent to the block unit. The nearby locations may be selected from at least one of a plurality of adjacent locations adjacent to the block unit or a plurality of non-adjacent locations not adjacent to the block unit.

[0205] Referring to FIGS. 1, 8, and 10, the encoder module 114 may determine multiple motion candidates from multiple candidate positions. The encoder module 114 may determine multiple neighboring blocks adjacent to the block unit based on the candidate positions. Each of the neighboring blocks may cover at least one of the candidate positions. The encoder module 114 may determine a template region adjacent to the block unit from the image frame and determine multiple neighboring regions from the image frame. Each of the neighboring regions may be adjacent to a corresponding one of the neighboring blocks. The encoder module 114 may determine a neighborhood cost value based on the template region and each of the multiple neighboring regions using a cost function to select a motion candidate for the block unit. The encoder module 114 may select multiple candidate blocks based on the neighborhood cost values ​​calculated by the neighboring regions. The candidate blocks may be reconstructed blocks reconstructed before reconstructing the block unit. If one of the candidate blocks does not have motion information, a motion candidate may not be determined based on the corresponding one of the selected neighboring positions. Additionally, the motion candidates may further include history-based candidates determined from a first-in-first-out (FIFO) table that stores multiple previous motion information of multiple previous blocks that were reconstructed before reconstructing the block unit.

[0206] Referring to FIGS. 1, 8, and 10, the encoder module 114 may determine multiple collocated blocks, each of which is included in one of one or more collocated frames, based on multiple motion candidates. Each of the multiple motion candidates may indicate a motion vector, list information, and frame information of a candidate block covering one of the multiple candidate positions. The encoder module 114 may determine one or more collocated frames based on at least one predetermined value. When one of multiple frame indexes of the frame information of a specific one of the candidate blocks is equal to one of the at least one predetermined value, a reference frame of the specific candidate block selected from a corresponding one of the reference lists indicated by the list information of the specific candidate block may be included in one or more collocated frames. The encoder module 114 may determine a motion block using a specific one of the motion candidates corresponding to the specific candidate block when the reference frame selected by the frame information of the specific motion candidate is included in one or more collocated frames. Each motion block may be considered a collocated block. In some embodiments, the encoder module 114 may determine a motion cost value for each of the motion blocks generated in different collocated frames based on the same motion vector, and the encoder module 114 may compare the motion cost values ​​of the motion blocks to select a collocated block.

[0207] 1, 8, and 10, the encoder module 114 may derive multiple sub-block-based temporal motion vector prediction (SbTMVP) candidates based on motion information of multiple collocated blocks. The encoder module 114 may determine whether the collocated blocks have motion information. The encoder module 114 may use the motion information of the collocated blocks to determine SbTMVP candidates that respectively correspond to one of the collocated blocks. When a block unit is predicted based on an SbTMVP candidate, the block unit may be divided into multiple sub-block units. Accordingly, each of the collocated blocks may also be divided into multiple collocated sub-blocks. Before reconstructing the image frame, the encoder module 114 may use the motion information of the collocated blocks to determine multiple first sub-block vectors Vs1 for the collocated sub-blocks as the collocated frame is reconstructed. Next, encoder module 114 may derive multiple second subblock vectors Vs2 for the subblock unit based on the multiple first subblock vectors Vs1. Furthermore, when a particular collocated subblock is included in a uni-predictive block, encoder module 114 may derive one second subblock vector for a particular subblock unit corresponding to the particular collocated subblock. Furthermore, when a particular collocated subblock is included in a bi-predictive block, encoder module 114 may derive two second subblock vectors for a particular subblock unit corresponding to the particular collocated subblock.

[0208] Referring to FIGS. 1, 8, and 10, the encoder module 114 may determine a reference cost value based on each of the multiple SbTMVP candidates. The encoder module 114 may determine a reference region including multiple reference subregions, each generated for a reference subblock corresponding to one of the collocated blocks. Because the reference subblocks corresponding to a specific collocated block may not be adjacent to each other, the reference subregion corresponding to the specific collocated block may be determined based on a portion of the M second subblock vectors Vs2. Some of the M second subblock vectors Vs2 may belong to a subblock unit located at the top boundary of the block unit or a subblock unit located at the left boundary of the block unit. The encoder module 114 may determine a reference cost value based on the template region and each of the multiple reference regions using a cost function.

[0209] Referring to Figures 1, 8, and 10, the encoder module 114 determines the arrangement of multiple SbTMVP candidates based on a first cost value and generates an adjusted SbTMVP list based on the arrangement. The encoder module 114 may determine the arrangement of the SbTMVP candidates based on a reference cost value and reorder the SbTMVP candidates based on the arrangement. The arrangement may further be determined by adjusting the new order of the SbTMVP candidates based on a diversity criterion. In some embodiments, the SbTMVP candidates may be divided into multiple candidate groups based on multiple division criteria. For example, the SbTMVP candidates may be divided into multiple candidate groups based on neighborhood location type, block size in block units, or at least one predetermined number. Furthermore, the encoder module 114 may collect only P SbTMVP candidates, and the amount of P SbTMVP candidates may be sufficient to distribute to the first p candidate groups.

[0210] 1, 8, and 10, the encoder module 114 may select one of a plurality of mode candidates in a candidate list, including at least one SbTMVP candidate, based on a mode selection method such as a cost function to predict the block unit. The encoder module 114 may predict the block unit based on at least one SbTMVP candidate in the candidate list to generate a plurality of predicted blocks. Next, the encoder module 114 may further compare a plurality of predicted samples included in each of the predicted blocks with a plurality of color components in the block unit. In this manner, the encoder module 114 may determine a block cost value for each predicted block and select a prediction candidate for the block unit from the mode candidates in the candidate list based on the block cost value. The mode selection method may be a rate-distortion optimization (RDO) process. Next, the encoder module 114 may determine a plurality of residual components by comparing the color components of the block unit with the predicted samples generated based on the prediction mode. Furthermore, the residual components and a prediction index may be encoded into a bitstream for a decoder device to reconstruct video data. The prediction index may indicate a selected one of the mode candidates in the candidate list. In some embodiments, the prediction index may be an index SbTMVP_idx for determining a selected one of the SbTMVP candidates when the candidate list includes only SbTMVP candidates.

[0211] 1, 8, and 10, the encoder module 114 may further reconstruct the block unit based on the prediction candidates in the candidate list to generate a reconstructed block including a plurality of reconstructed samples, which may be used as references for predicting a plurality of subsequent blocks in the video data.

[0212] Therefore, encoder module 114 may also use method 800 to predict and reconstruct block units for encoding video data into a bitstream. Because encoder module 114 and decoder module 124 may use the same method 800, encoder module 114 and decoder module 124 may generate the same reordering list to reduce the number of bits in the bitstream.

[0213] Method M2, which is a method for encoding / decoding video data and is generated by combining method 300 and method 800, may be executed by the first electronic device 110. In method M2, the encoder module 114 may encode video data based on blocks 310-360 of method 300. Next, the encoder module 114 may select a portion of the collocated block based on the collocated cost value of the collocated block. After the portion of the collocated block is selected, the encoder module 114 may encode the video data based on blocks 860-890 of method 800. In this manner, the encoder module 114 may derive SbTMVP candidates based on motion information of the portion of the collocated block to reconstruct a block unit.

[0214] The disclosed embodiments are to be considered in all respects as illustrative and not restrictive, and it should be understood that the disclosure is not limited to the particular disclosed embodiments, but that many rearrangements, modifications, and substitutions are possible without departing from the scope of the disclosure. [Brief explanation of the drawings]

[0215] Aspects of the present disclosure are best understood from the following detailed disclosure and corresponding figures, in which: Various features are not drawn to scale and dimensions of various features may be arbitrarily increased or decreased for clarity of discussion. [Figure 1]FIG. 1 is a block diagram of a system configured to encode and decode video data according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a block diagram of a decoder module of the second electronic device illustrated in FIG. 1 according to an embodiment of the present disclosure. [Figure 3] FIG. 3 shows a flowchart of a method for decoding video data by an electronic device according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram illustrating an example implementation of a block and multiple candidate locations according to one embodiment of the present disclosure. [Figure 5] 5A, 5B, and 5C are diagrams illustrating exemplary implementations of a template region and different adjacent regions according to one embodiment of the present disclosure. [Figure 6] FIG. 6 is an illustrative diagram illustrating an example implementation of an image frame having block units and template regions, and a collocated frame having collocated blocks and collocated regions, according to one embodiment of the present disclosure. [Figure 7] FIG. 7 is an illustrative diagram showing an example implementation of an image frame having block units and template regions, and a collocated frame having collocated blocks, shifted blocks, and collocated regions, according to one embodiment of the present disclosure. [Figure 8] FIG. 8 shows a flowchart of a method for decoding video data by an electronic device according to an embodiment of the present disclosure. [Figure 9] 9A and 9B are diagrams illustrating exemplary implementations of template regions and reference regions according to embodiments of the present disclosure. [Figure 10] FIG. 10 is a block diagram of an encoder module of the first electronic device shown in FIG. 1 according to an embodiment of the present disclosure.

Claims

1. 1. A method for decoding video data by an electronic device, comprising: receiving the video data; determining block units from an image frame according to the video data; determining a plurality of candidate locations including at least one non-adjacent location that is not adjacent to the block unit; determining a plurality of motion candidates from the plurality of candidate positions; selecting a plurality of collocated blocks, each of which is comprised in one of one or more collocated frames comprised in the video data, based on the plurality of motion candidates; determining a first cost value for each of the plurality of collocated blocks; determining at least one sub-block-based temporal motion vector prediction (SbTMVP) candidate based on the plurality of motion candidates and the plurality of first cost values; reconstructing the block unit based on the at least one SbTMVP candidate; determining, from the image frame, a plurality of neighboring blocks adjacent to the block unit based on the plurality of candidate locations, wherein at least one of the plurality of neighboring blocks covers one of the at least one non-neighboring location; determining, from the image frame, a template region adjacent to the block unit and a plurality of neighboring regions each adjacent to one of the plurality of neighboring blocks; determining a second cost value based on the template region and each of the plurality of neighboring regions; selecting a plurality of candidate blocks based on the plurality of second cost values; and determining the plurality of motion candidates from the plurality of candidate blocks; A method comprising:

2. The method of claim 1 , wherein the plurality of candidate locations further includes at least one of a plurality of adjacent locations adjacent to the block unit.

3. A method for decoding video data by an electronic device, comprising: receiving the video data; determining block units from an image frame according to the video data; determining a plurality of candidate locations including at least one non-adjacent location that is not adjacent to the block unit; determining a plurality of motion candidates from the plurality of candidate positions; selecting a plurality of collocated blocks, each of which is comprised in one of one or more collocated frames comprised in the video data, based on the plurality of motion candidates; determining a first cost value for each of the plurality of collocated blocks; determining at least one sub-block-based temporal motion vector prediction (SbTMVP) candidate based on the plurality of motion candidates and the plurality of first cost values; and reconstructing the block unit based on the at least one SbTMVP candidate; each of the plurality of motion candidates indicates a motion vector, list information, and frame information of a candidate block covering one of the plurality of candidate positions; A method wherein, if a frame index in the frame information of a particular candidate block is equal to one of at least one predetermined value, a reference frame of the particular candidate block selected from a reference list indicated by list information of the particular candidate block is included in the one or more collocated frames.

4. The method of claim 3 , wherein one of the plurality of collocated blocks is selected from the reference frame for the particular candidate block based on the motion vector and a corresponding one of the plurality of candidate locations.

5. 4. The method of claim 3, wherein if the frame index in the frame information of the particular candidate block differs from at least one predetermined value, the reference frame of the particular candidate block is excluded from the one or more collocated frames.

6. A method for decoding video data by an electronic device, comprising: receiving the video data; determining block units from an image frame according to the video data; determining a plurality of candidate locations including at least one non-adjacent location that is not adjacent to the block unit; determining a plurality of motion candidates from the plurality of candidate positions; selecting a plurality of collocated blocks, each of which is comprised in one of one or more collocated frames comprised in the video data, based on the plurality of motion candidates; determining a first cost value for each of the plurality of collocated blocks; determining at least one sub-block-based temporal motion vector prediction (SbTMVP) candidate based on the plurality of motion candidates and the plurality of first cost values; and reconstructing the block unit based on the at least one SbTMVP candidate; each of the plurality of motion candidates indicates a motion vector, list information, and frame information of a candidate block covering one of the plurality of candidate positions; If the particular candidate block is a bi-predictive block, at least one reference frame for the particular candidate block is determined from two reference lists for the particular candidate block; A method wherein, if one of two frame indices included in the frame information of the particular candidate block is equal to one of at least one predetermined value, the at least one reference frame is included in the one or more collocated frames.

7. adding said at least one SbTMVP candidate to a candidate list; selecting a predicted candidate from the candidate list based on a prediction index; and reconstructing the block units based on a prediction mode; The method of claim 1 further comprising:

8. The method of claim 1 , wherein the plurality of collocated blocks is selected from a plurality of collocated frames.

9. determining a template region adjacent to the block unit from the image frame, and determining a plurality of collocated regions from the one or more collocated frames, each of the collocated regions being adjacent to one of the plurality of collocated blocks; and determining the first cost value between the template region and each of the plurality of collocated regions; The method of claim 1 further comprising:

10. 1. An electronic device for decoding video data, comprising: at least one processor; coupled to the at least one processor and, when executed by the at least one processor, causing the electronic device to receiving the video data; determining block units from an image frame according to the video data; determining a plurality of candidate locations including at least one non-adjacent location that is not adjacent to the block unit; determining a plurality of motion candidates from the plurality of candidate positions; selecting a plurality of collocated blocks, each of which is included in one of one or more collocated frames included in the video data, based on the plurality of motion candidates; determining a first cost value for each of the plurality of collocated blocks; determining at least one sub-block-based temporal motion vector prediction (SbTMVP) candidate based on the plurality of motion candidates and the plurality of first cost values; reconstructing the block unit based on the at least one SbTMVP candidate; determining a plurality of neighboring blocks adjacent to the block unit from the image frame based on the plurality of candidate locations; determining a template region adjacent to the block unit and a plurality of neighboring regions each adjacent to one of the plurality of neighboring blocks from the image frame; determining a second cost value based on the template region and each of the plurality of neighboring regions; selecting a plurality of candidate blocks based on the plurality of second cost values; determining the plurality of motion candidates from the plurality of candidate blocks; and At least one of the plurality of neighboring blocks covers one of the at least one non-adjacent location.

11. The electronic device of claim 10 , wherein the plurality of candidate locations further includes at least one of a plurality of adjacent locations adjacent to the block unit.

12. An electronic device for decoding video data, comprising: at least one processor; coupled to the at least one processor and, when executed by the at least one processor, causing the electronic device to receiving the video data; determining block units from an image frame according to the video data; determining a plurality of candidate locations including at least one non-adjacent location that is not adjacent to the block unit; determining a plurality of motion candidates from the plurality of candidate positions; selecting a plurality of collocated blocks, each of which is included in one of one or more collocated frames included in the video data, based on the plurality of motion candidates; determining a first cost value for each of the plurality of collocated blocks; determining at least one sub-block-based temporal motion vector prediction (SbTMVP) candidate based on the plurality of motion candidates and the plurality of first cost values; and at least one storage device that stores a plurality of computer-executable instructions that cause the block unit to be reconstructed based on the at least one SbTMVP candidate; each of the plurality of motion candidates indicates a motion vector, list information, and frame information of a candidate block covering one of the plurality of candidate positions; An electronic device, wherein if a frame index in the frame information of a particular candidate block is equal to one of at least one predetermined value, a reference frame of the particular candidate block selected from a reference list indicated by list information of the particular candidate block is included in the one or more collocated frames.

13. The electronic device of claim 12 , wherein one of the plurality of collocated blocks is selected from the reference frame for the particular candidate block based on the motion vector and a corresponding one of the plurality of candidate locations.

14. 13. The electronic device of claim 12, wherein the reference frame of the particular candidate block is excluded from the one or more collocated frames if the frame index in the frame information of the particular candidate block differs from at least one predetermined value.

15. An electronic device for decoding video data, comprising: at least one processor; coupled to the at least one processor and, when executed by the at least one processor, causing the electronic device to receiving the video data; determining block units from an image frame according to the video data; determining a plurality of candidate locations including at least one non-adjacent location that is not adjacent to the block unit; determining a plurality of motion candidates from the plurality of candidate positions; selecting a plurality of collocated blocks, each of which is included in one of one or more collocated frames included in the video data, based on the plurality of motion candidates; determining a first cost value for each of the plurality of collocated blocks; determining at least one sub-block-based temporal motion vector prediction (SbTMVP) candidate based on the plurality of motion candidates and the plurality of first cost values; and at least one storage device that stores a plurality of computer-executable instructions that cause the block unit to be reconstructed based on the at least one SbTMVP candidate; each of the plurality of motion candidates indicates a motion vector, list information, and frame information of a candidate block covering one of the plurality of candidate positions; if a particular candidate block is a bi-predictive block, at least one reference frame for the particular one of the candidate blocks is determined from two reference lists for the particular candidate block; An electronic device, wherein if one of two frame indexes included in the frame information of the particular candidate block is equal to one of at least one predetermined value, the at least one reference frame is included in the one or more collocated frames.

16. The plurality of computer-executable instructions, when executed by the at least one processor, further cause the electronic device to: adding the at least one SbTMVP candidate to a candidate list; selecting a predicted candidate from the candidate list based on a prediction index; The electronic device of claim 10 , configured to reconstruct the block units based on a prediction mode.

17. The electronic device of claim 10 , wherein the plurality of collocated blocks are selected from a plurality of collocated frames.

18. The plurality of computer-executable instructions, when executed by the at least one processor, further cause the electronic device to: determining a template region adjacent to the block unit from the image frame; and determining a plurality of collocated regions from the one or more collocated frames, each of the collocated regions being adjacent to one of the plurality of collocated blocks; The electronic device of claim 10 , further comprising: determining the first cost value between the template region and each of the plurality of collocated regions.

Images