Searching apparatus and method of template matching prediction technology of atlas image

Abstract

The image encoding / decoding method, device, and recording medium of the present disclosure may comprise the steps of: configuring a merge candidate list; rearranging the merge candidate list; generating a template matching candidate list on the basis of the merge candidate list; and performing a template matching search on the basis of an initial point of the template matching candidate list.

Description

Search device and method for template matching prediction technology of atlas images

[0001] The present disclosure can be utilized in the fields of encoding / decoding methods, devices, and recording media of atlas images.

[0002] MIV (MPEG Immersive Video) is a compression standard for encoding / decoding and rendering used to reproduce images that will be seen when moving in 3DoF (Degree of Freedom), 3DoF+, and 6DoF degrees of freedom.

[0003] 3DoF+ can support motion parallax for very limited movements, such as slight head movements in a seated environment, by synthesizing multiple images including omnidirectional scenes and multiple virtual viewpoint images based on them.

[0004] 6DoF can support degrees of freedom for six directions of movement (left-right, up-and-down, forward-backward, pitch, roll, yaw).

[0005] The MIV encoding standard technology aims to provide an interface for viewing immersive video and reproduce high-quality viewport images, along with the compression efficiency of traditional video coding.

[0006] TM (Template Matching) technology may require high complexity when encoding / decoding Atlas images because it must perform a sub-pel search process for all four MVP (Motion Vector Prediction) candidates in the TMCL (Template Matching Candidate List).

[0007] The present disclosure aims to reduce the complexity of a video compression codec by reducing the number of applications by applying a search pattern according to specific conditions.

[0008] The video encoding / decoding method, device, and recording medium of the present disclosure may include a step of configuring a merge candidate list, a step of rearranging the merge candidate list, a step of generating a template matching candidate list based on the merge candidate list, and a step of performing a template matching search based on an initial point of the template matching candidate list.

[0009] In the video encoding / decoding method, device and recording medium of the present disclosure, the step of constructing the merge candidate list can be performed using up to 10 MVPs (Motion Vector Prediction).

[0010] In the video encoding / decoding method, device, and recording medium of the present disclosure, the composition of the merge candidate list can be performed in the order of spatial MVP, temporal MVP, non-adjacent spatial MVP, history-based MVP, combination MVP, and zero MVP.

[0011] In the video encoding / decoding method, device, and recording medium of the present disclosure, the template matching candidate list can be generated using candidates having index values ​​0 to 3 of the merge candidate list.

[0012] In the video encoding / decoding method, device and recording medium of the present disclosure, the template matching candidate list may have up to four candidates.

[0013] In the video encoding / decoding method, device and recording medium of the present disclosure, the initial point may be the first candidate of the template matching candidate list.

[0014] In the video encoding / decoding method, device and recording medium of the present disclosure, the template matching search is performed based on any one of the search patterns, and the search pattern can be determined by the search shape and search range.

[0015] In the video encoding / decoding method, device and recording medium of the present disclosure, the search pattern and the search shape may include a cross search and a diamond search.

[0016] In the video encoding / decoding method, device and recording medium of the present disclosure, the search range may include 4-pel, integer-pel, half-pel, quarter-pel and 1 / 8-pel.

[0017] In the video encoding / decoding method, device and recording medium of the present disclosure, the search pattern can be determined based on a comparison of SAD values ​​for the same search range for each search shape.

[0018] In the video encoding / decoding method, device and recording medium of the present disclosure, the search pattern can be determined based on a comparison of SAD values ​​of two search ranges for the same search shape.

[0019] The present disclosure can reduce the complexity and improve the efficiency of a video compression codec by reducing the number of applications by applying a search pattern according to specific conditions.

[0020] Figure 1 illustrates the overall structure of TMIV (Test model for MIV).

[0021] Figure 2 illustrates the differences between natural images and Atlas images generated from TMIV.

[0022] Figure 3 shows an example when up to 10 candidates are used in the MCL (Merge candidate list).

[0023] Figure 4 shows an example of a changed MCL after ARMC-TM is applied.

[0024] Figure 5 illustrates an example TMCL constructed using the generated MCL candidates.

[0025] Figure 6 illustrates an example showing a template of a reference block for the first MVP candidate among the TMCL candidates.

[0026] Figure 7 illustrates an example of a search pattern.

[0027] Figure 8 shows examples of cross search and diamond search among search shapes of search patterns.

[0028] Figure 9 illustrates an example that performs only a diamond search-shaped search pattern.

[0029] Figure 10 shows an example of the range of sub-pels within the search range.

[0030] FIG. 11 illustrates a flowchart of a method for performing the TM prediction technique of the present disclosure.

[0031] FIG. 12 illustrates a device for performing the TM prediction technique of the present disclosure.

[0032] The present invention is susceptible to various modifications and embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present invention. Throughout the description of each drawing, similar reference numerals have been used to designate similar components.

[0033] While terms such as "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component." The term "and / or" includes a combination of multiple related items described herein or any of multiple related items described herein.

[0034] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0035] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0036] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. Hereinafter, identical components in the drawings will be designated by the same reference numerals, and redundant descriptions of identical components will be omitted.

[0037]

[0038] Figure 1 illustrates the overall structure of TMIV (Test model for MIV).

[0039] Currently, TMIV performs preprocessing on re-view video to generate a small number of atlas images, and can perform compression using traditional video codecs such as HEVC (High Efficiency Video Coding) and VVC (Versatile Video Coding).

[0040] Figure 2 illustrates the differences between natural images and Atlas images generated from TMIV.

[0041] During the standardization process of traditional video codecs, test images (test sequences) used as Common Test Conditions (CTC) are mostly images obtained from nature, and there is a problem in that they do not consider the characteristics of the Atlas images generated from TMIV.

[0042] Figure 3 shows an example when up to 10 candidates are used in the MCL (Merge candidate list).

[0043] Template Matching (TM) technology can derive motion vectors for inter-frame prediction using templates that have been encoded / decoded around the currently encoded block. The template can have an arbitrary size of NxM, where N and M can be defined as positive integers or have the same value. For example, NxM can have a size of 3x3.

[0044] A motion vector with a small SAD cost between the template whose encoding / decoding is complete and the template of the reference block can be found and used as the motion vector of the current block.

[0045] Additionally, this technology can be applied to Advanced Motion Vector Prediction (AMVP) mode and MERGE mode. However, to facilitate the explanation of this technology, the TM technology for MERGE mode will be described below.

[0046] Hereinafter, the process of the present disclosure for performing the TM technology may be composed of several steps.

[0047] The first step may include constructing a Merge candidate list (MCL).

[0048] For example, the construction of an MCL can be performed by populating the MCL with up to 10 MVPs in the following order: spatial MVP, temporal MVP, non-adjacent spatial MVP (NA-SMVP), history-based MVP, pair-wise MVP, and zero MVP. Here, the maximum number of MVP candidates in the MCL can be signaled from the bitstream or implicitly determined based on the characteristics of the current block (size, shape, partitioning method, etc.).

[0049] Additionally, the way to populate the MCL using each MVP can be the same as the way to populate the general MCL.

[0050] The second step may include performing ARMC-TM (Adaptive Reordering Merge Candidate with Template Matching) on ​​the generated MCL.

[0051] In the above step, the candidates of the generated MCL can be rearranged.

[0052] Figure 4 shows an example of a changed MCL after ARMC-TM is applied.

[0053] ARMC-TM may be a technology that performs realignment based on template matching by utilizing the motion information of the generated MCL.

[0054] The third step may include generating a TM candidate list (TMCL).

[0055] Specifically, the above step can be performed by generating a TMCL using the generated MCL candidate.

[0056] For example, TMCL can be generated using the 0th to 3rd indices among the generated MCL candidates.

[0057] For example, a TMCL can be generated using MCL candidate indices signaled from a bitstream.

[0058] The generated TMCL can use up to four MVP candidates. The maximum number of MVP candidates in the TMCL can be signaled from the bitstream or implicitly determined based on the characteristics of the current block (size, shape, partitioning method, maximum number of MVP candidates, etc.).

[0059] Figure 5 illustrates an example TMCL constructed using the generated MCL candidates.

[0060] The fourth step may include a step of performing a TM search process based on the Initial point.

[0061] Specifically, the above step can perform a TM search process using the MVP corresponding to the index of TMCL as an initial point.

[0062] Figure 6 illustrates an example of a template for a reference block for the first MVP candidate among TMCL candidates. Specifically, Figure 6 illustrates an example of a TM search process using the first MVP candidate as an initial point.

[0063] Figure 7 illustrates an example of a search pattern.

[0064] Search patterns can be determined by search shape (Cross search, Diamond search, etc.) and search range (4-pel, integer-pel, half-pel, quarter-pel, 1 / 8-pel, etc.).

[0065] Referring to Fig. 7, it can be seen that the search shapes include only Cross search and Diamond search, and each search shape includes a search range of 4-pel, Full-pel, Half-pel, Quarter-pel, and 1 / 8-pel.

[0066] For example, the search range for performing template matching based on the initial point can be any one of 4-pel, integer-pel, half-pel, quarter-pel, and 1 / 8-pel, as shown in Fig. 7.

[0067] Figure 8 shows examples of cross search and diamond search among search shapes of search patterns.

[0068] In the case of a cross search, the search can be in the up, down, left, and right directions based on the initial point. In the case of a diamond search, the search can be in the shape of a diamond centered on the initial point.

[0069] In the above step, the final predictor can be generated after performing the process for all combinations of the four MVP candidates for the search pattern.

[0070] TM technology may be a method of estimating a motion vector by comparing the SAD cost between reference templates based on templates that have been completed encoding / decoding around the block currently being encoded, in order to implicitly derive a motion vector for the current encoding block in the encoder / decoder.

[0071] At this time, the search pattern for calculating the SAD cost can be determined by the shape and search range of the search pattern. The shape of the search pattern can include a cross search, a diamond search, a TZ search, a full search, etc. The shape of the search pattern can be pre-defined in the device, implicitly determined based on the characteristics of the current block, or determined by search shape information included in the search pattern information signaled from the bitstream.

[0072] The search range may include 4-pel, 1-pel, 1 / 2-pel, 1 / 4-pel, 1 / 8-pel, and 1 / 16-pel. The search range may be predefined in the device, implicitly determined based on the characteristics of the current block, or determined by search range information included in search pattern information signaled from the bitstream.

[0073] In one embodiment, when calculating the SAD cost for the current encoding block, cross search and diamond search are used as search shapes, and 1-pel, 1 / 2-pel, 1 / 4-pel, and 1 / 8-pel are used as search ranges, and searches (or SAD cost calculations) can be performed for all possible combinations of search shapes and search ranges.

[0074] In contrast, when calculating the SAD cost, searches (or SAD cost calculations) may be performed only for some search patterns based on comparison of SAD values ​​for the same search range by search shape.

[0075] For example, if the SAD value calculated through the diamond search pattern of the search range integer-pel is smaller than the SAD value of the cross search pattern of integer-pel, the search (or SAD cost calculation) for the sub-pel (1 / 2-pel, 1 / 4-pel, 1 / 8-pel, etc.) can only be performed through the diamond search-shaped search pattern.

[0076] Additionally, searches (or SAD cost calculations) may be performed only for some search patterns based on comparison of SAD values ​​of different search ranges for the same search shape.

[0077] For example, in a specific search shape, if the SAD value of the search range integer-pel is smaller than the SAD value of the search range quarter-pel, then only the search (or calculation) for the half-pel can be performed in the search pattern of the search shape.

[0078] For example, in a specific search shape, if the integer-pel SAD value is greater than the SAD value of the search range integer-pel and the SAD value of the search range quarter-pel, then only the search (or calculation) for 1 / 8-pel can be performed in the search pattern of the search shape.

[0079] For example, for a specific search shape, if the SAD value of the search range integer-pel is smaller than the search range half-pel by a factor of N or more, then only the search (or calculation) for the four-pel can be performed in the search pattern of the search shape. Here, N can have a value greater than or equal to 0. The value of N can be pre-defined in the device, implicitly determined based on the characteristics of the current block, or determined by search range information included in the search pattern information signaled from the bitstream.

[0080] Figure 9 illustrates an example that performs only a diamond search-shaped search pattern.

[0081] Figure 10 shows an example of the range of sub-pels within the search range.

[0082] FIG. 11 illustrates a flowchart of a method for performing the TM prediction technique of the present disclosure.

[0083] The TM prediction technology of the present disclosure may include a step of constructing a Merge Candidate List (S1101), a step of rearranging the Merge Candidate List (S1102), a step of generating a Template Matching Candidate List (TMCL) based on the Merge Candidate List (S1103), and a step of performing a template matching search based on an initial point of the Template Matching Candidate List (S1104). The specific details of each step are omitted as they have been described above.

[0084] FIG. 12 illustrates a device for performing the TM prediction technique of the present disclosure.

[0085] It may include a merge candidate list construction unit (1201) that performs a step of constructing a merge candidate list, a merge candidate list rearrangement unit (1202) that performs a step of rearranging the merge candidate list, a template matching candidate list generation unit (1203) that performs a step of generating a template matching candidate list (TMCL) based on the merge candidate list, and a template matching search unit (1204) that performs a step of performing a template matching search based on an initial point of the template matching candidate list.

[0086]

[0087] While the exemplary methods of this disclosure are presented as a series of operations for clarity of description, this is not intended to limit the order in which the steps are performed, and individual steps may be performed simultaneously or in different orders, if desired. To implement a method according to this disclosure, additional steps may be included in addition to the steps illustrated, some steps may be excluded and the remaining steps may be included, or some steps may be excluded and additional steps may be included.

[0088] The various embodiments of the present disclosure are not intended to list all possible combinations but rather to illustrate representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combinations of two or more.

[0089] Additionally, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In the case of hardware implementation, the embodiments may be implemented by one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), general processors, controllers, microcontrollers, microprocessors, etc.

[0090] The scope of the present disclosure includes software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that cause operations according to the methods of various embodiments to be executed on a device or a computer, and a non-transitory computer-readable medium having such software or instructions stored thereon and executable on the device or computer.

[0091] The image encoding / decoding method, device, and recording medium of the present disclosure may be industrially applicable in the field of encoding / decoding atlas images.

Claims

1. Step of creating a merge candidate list; A step of reordering the above merge candidate list; A step of generating a template matching candidate list based on the above merge candidate list; and A video encoding method, characterized by comprising a step of performing a template matching search based on an initial point of the template matching candidate list.

2. In paragraph 1, A video encoding method, characterized in that the step of constructing the above merge candidate list is performed using up to 10 MVPs (Motion Vector Prediction).

3. In paragraph 1, A video encoding method, characterized in that the composition of the above merge candidate list is performed in the order of spatial MVP, temporal MVP, non-adjacent spatial MVP, history-based MVP, combination MVP, and zero MVP.

4. In paragraph 1, A video encoding method, characterized in that the template matching candidate list is generated using candidates having index values ​​0 to 3 of the merge candidate list.

5. In paragraph 1, A video encoding method, characterized in that the above template matching candidate list has a maximum of four candidates.

6. In paragraph 1, A video encoding method, characterized in that the initial point is the first candidate in the template matching candidate list.

7. In paragraph 1, The above template matching search is performed based on any one of the search patterns, An image encoding method, characterized in that the search pattern is determined by a search shape and a search range.

8. In paragraph 7, A method for encoding an image, wherein the search shape includes a cross search and a diamond search.

9. In paragraph 8, An image encoding method, characterized in that the above search range includes 4-pel, integer-pel, half-pel, quarter-pel and 1 / 8-pel.

10. In paragraph 9, A method for encoding an image, wherein the above search pattern is determined based on a comparison of SAD values ​​for the same search range for each search shape.

11. In paragraph 9, An image encoding method, characterized in that the search pattern is determined based on a comparison of SAD values ​​of two search ranges for the same search shape.

12. Step of creating a merge candidate list; A step of reordering the above merge candidate list; A step of generating a template matching candidate list based on the above merge candidate list; and A method for decoding an image, characterized by comprising a step of performing a template matching search based on an initial point of the template matching candidate list.

13. In paragraph 12, The above template matching search is performed based on any one of the search patterns, An image decoding method, characterized in that the search pattern is determined by a search shape and a search range.

14. In paragraph 13, A method for encoding an image, wherein the above search pattern is determined based on a comparison of SAD values ​​for the same search range for each search shape.

15. In a computer-readable recording medium storing a bitstream generated by an image encoding method, The above image encoding method is, Steps to construct a list of merge candidates; A step of reordering the above merge candidate list; A step of generating a template matching candidate list based on the above merge candidate list; and A computer-readable recording medium characterized by comprising a step of performing a template matching search based on an initial point of the template matching candidate list.

Images