Method and apparatus for video encoding and decoding on basis of geometric partitioning prediction
Patent Information
- Application Number
- HK62026126836
- Authority / Receiving Office
- HK · HK
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-08-31
- Filing Date
- 2026-07-29
- Publication Date
- 2026-09-11
Smart Images

Figure 00000000_0000_ABST 
Figure 00000000_0001_ABST
Abstract
Description
(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202480046129.8 (22) Application Date 2024.06.20 (30) Priority Data 10-2023-0089381 2023.07.10 KR 10-2023-0115378 2023.08.31 KR (85) PCT International Application Entering National Phase Date 2026.01.08 (86) PCT International Application Application Data PCT / KR2024 / 008519 2024.06.20 (87) PCT International Application Publication Data WO2025 / 014115 KO 2025.01.16 (71) Applicant: Jiawen Group Co., Ltd. Address: South Korea Applicant: Sejong University Industry-Academia Collaboration Group (72) Inventors: Yoo Chang-woo, Lee Jin-young (74) Patent Agency: Beijing Pinyuan Patent Agency Co., Ltd. 11332 Patent Attorneys: Lü Lin, Tian Ying-ai (51) Int.Cl. H04N 19 / 119 (2006.01) H04N 19 / 157 (2006.01) H04N 19 / 176 (2006.01) H04N 19 / 105 (2006.01) H04N 19 / 139 (2006.01) H04N 19 / 70 (2006.01) (54) Invention Title: A Video Encoding and Decoding Method and Apparatus Based on Geometric Segmentation Prediction (57) Abstract: This invention provides a video encoding and decoding method and apparatus based on geometric segmentation prediction, which can be used in the above-mentioned video encoding and decoding fields. A video decoding method based on geometric partitioning prediction according to an embodiment of the present invention may include the following steps: reading geometric partitioning mode (GPM) information about the geometric partitioning of blocks from an encoded bitstream; determining a first partitioning region and a second partitioning region based on the GPM information; reading first prediction information from the encoded bitstream; reading second prediction information from the encoded bitstream; and generating a first prediction block about the first partitioning region based on the first prediction information, generating a second prediction block about the second partitioning region based on the second prediction information, and combining the first prediction block and the second prediction block to generate a final prediction block, wherein the first prediction information and the second prediction information are different from each other.Claims (3 pages), Description (22 pages), Drawings (10 pages), CN 121488465 A, 2026.02.06, CN 1 21 48 84 65 A. 1. A decoding method, as a video decoding method based on geometric segmentation prediction, characterized in that it includes: a step of reading geometric segmentation pattern information about the geometric segmentation of blocks from an encoded bitstream; a step of determining a first segmentation region and a second segmentation region based on the geometric segmentation pattern information; a step of reading first prediction information from the encoded bitstream; a step of reading second prediction information from the encoded bitstream; and a step of generating a first prediction block about the first segmentation region based on the first prediction information, generating a second prediction block about the second segmentation region based on the second prediction information, and combining the first prediction block and the second prediction block to generate a final prediction block, wherein the first prediction information and the second prediction information are different from each other. 2. The decoding method according to claim 1, wherein the step of generating the final prediction block includes: determining a boundary region including the boundary lines of the first segmentation region and the second segmentation region based on the geometric segmentation mode information; generating the boundary region prediction block; generating the final prediction block by combining the first prediction block, the second prediction block, and the boundary region prediction block; and storing boundary region information about the boundary region prediction block. 3. The decoding method according to claim 2, wherein the boundary region prediction block is generated based on the average of the weighted values of the first prediction block and the second prediction block. 4. The decoding method according to claim 3, wherein the weighted value is determined based on at least one of the following: the first prediction information, the second prediction information, the prediction information of the surrounding block, whether the prediction information of the surrounding block is intra-frame prediction information, the distance between the surrounding block and the generated boundary region, and the reference vector information of the surrounding block. 5. The decoding method according to claim 2, wherein the first prediction information includes first reference vector information, the second prediction information does not include reference vector information, and the boundary region information is generated based on the first reference vector. 6. The decoding method according to claim 2, wherein the first prediction information regarding the first segmented region includes first reference vector information, the second prediction information regarding the second segmented region includes second reference vector information, and the boundary region information is generated based on at least one of the first prediction information and the second prediction information. 7. The decoding method according to claim 6, wherein the first prediction information pertains to an intra-frame block copy mode, the first reference vector represents a block vector, and the boundary region information is generated based on the first prediction information.8. The decoding method according to claim 7, wherein the first prediction information includes a first index of a first intra-frame block copy reference vector candidate list. 9. The decoding method according to claim 7, wherein the second prediction information is about an intra-frame block copy mode, the second reference vector represents a block vector, and the boundary region information is generated based on the first prediction information and the second prediction information. 10. The decoding method according to claim 9, wherein the first prediction information includes a first index of a first intra-frame block copy reference vector candidate list, the second prediction information includes a second index of a second intra-frame block copy reference vector candidate list, and the boundary region information is generated based on third index information, wherein the third index information is determined based on at least one of the first index and the second index. 11. The decoding method according to claim 4, wherein the first prediction information is about an inter-frame mode, the first reference vector represents a motion vector, and the boundary region information is generated based on the first prediction information. 12. The decoding method according to claim 11, wherein the first prediction information includes a first index of a first reference vector candidate list based on a merged mode. 13. The decoding method according to claim 11, wherein the second prediction information is about the inter-frame mode, the second reference vector represents a motion vector, and the boundary region information is generated based on the first prediction information and the second prediction information. 14. The decoding method according to claim 13, wherein the first prediction information includes a first index based on a first reference vector candidate list under a merging mode, the second prediction information includes a second index based on a second reference vector candidate list under a merging mode, and the boundary region information is generated based on third index information, wherein the third index information is determined based on at least one of the first index and the second index. 15. The decoding method according to claim 13, wherein the first prediction information includes a bidirectional reference vector, the second prediction information includes a unidirectional reference vector, and the boundary region information is generated based on the first prediction information. 16. The decoding method according to any one of claims 6, 7, or 11, wherein the boundary region information is generated based on a reference vector obtained by weighted averaging of the first reference vector and the second reference vector.17. An encoding method, as a video encoding method based on geometric segmentation prediction, characterized in that it includes: determining a first segmentation region and a second segmentation region based on block geometric segmentation information; determining first prediction information about the first segmentation region; determining second prediction information about the second segmentation region; and encoding the first prediction information, the second prediction information, and the combined information into a bitstream, wherein the first prediction information and the second prediction information are different from each other. 18. The encoding method according to claim 17, characterized in that it further includes: setting a boundary region including the boundary line of the first segmentation region and the second segmentation region; and storing boundary region information about the boundary region prediction block. 19. The encoding method according to claim 18, characterized in that, the first prediction information includes a first index of a first intra-frame block copy reference vector candidate list, the second prediction information includes a second index of a second intra-frame block copy reference vector candidate list, and the boundary region information includes third index information determined based on at least one of the first index and the second index. 20. A decoder device, as a video decoder device based on geometric segmentation prediction, characterized in that it comprises: a bitstream parsing unit, configured to read geometric segmentation mode information, first prediction information and second prediction information about the geometric segmentation of blocks from an encoded bitstream; a geometric segmentation prediction unit, configured to determine a first segmentation region and a second segmentation region based on the geometric segmentation mode information, generate a first prediction block about the first segmentation region based on the first prediction information, generate a second prediction block about the second segmentation region based on the second prediction information, and generate a final prediction block by combining the first prediction block and the second prediction block; and a memory, configured to store decoded image information including the final prediction block, wherein the first prediction information and the second prediction information are different from each other. Claims 3 / 3 Page 4 CN 121488465 A A Video Encoding and Decoding Method and Apparatus Based on Geometric Segmentation Prediction Technical Field
[0001] The present invention relates to the field of encoding and decoding of digital video, and more specifically, to methods for encoding and decoding digital video, methods for recording such data, and components, apparatus and systems for implementing such methods.
[0002] This invention can be in the same technical field as at least one known digital video compression technology standard (e.g., MPEG-2, MPEG-4 Video, H.263, H.264 / AVC, H.265 / HEVC, H.266 / VVC, VC-1, AVI, QuickTime, VP-9, VP-10, and Motion JPEG), or it can be used to improve the inherent efficiency of such standards, or it can be used to improve or replace such standards. Background Art
[0003] Digital video encoding and decoding are widely used in various digital video application fields. For example, digital television broadcasting, video transmission through communication networks, video calls, video conversations, video chat, video content recorded and played using optical discs (including VCD (video compact disc) / DVD (digital versatile disc) / Blu-Ray disc), the entire process of video content production, editing, collection and distribution, and video shooting devices and cameras used in video shooting and recording activities (including personal, commercial, industrial and security monitoring purposes, etc.) all rely on video encoding and decoding technology.
[0004] Therefore, physical devices that can be called digital video encoders and decoders can constitute part of a wide range of devices related to the generation, recording, and playback of digital video, including digital television, digital broadcasting systems, wireless broadcasting systems, laptops, desktop computers, tablets, e-book readers, digital cameras, digital video recording devices, digital multimedia playback devices, video game devices / terminals / control consoles, mobile phones (including smartphones) with multimedia playback capabilities, devices for video conferencing, and others.
[0005] The aforementioned digital video encoders and decoders can be implemented using digital video compression standards widely used and understood by those skilled in the art. These digital video compression standards may include at least one known compression standard, such as those named MPEG-2, MPEG-4 Video, H.263, H.264 / AVC, H.265 / HEVC, H.266 / VVC, VC-1, AVI, QuickTime, VP-9, VP-10, and Motion JPEG.
[0006] Video encoders and decoders can more efficiently encode or decode digital video information by conforming to the aforementioned standards, or by improving or modifying the aforementioned standards. The attempt to modify the above standards may also lead to the creation of new standards.In a well-known case, there exists an enhanced compression model (ECM), developed by the Joint Video Experts Team (JVET), a joint international standardization organization of ISO, IEC, and ITU-T, aimed at improving and replacing the existing H.266 / VVC standard.
[0007] Problems to be Solved by the Invention
[0008] As is well known, digital video, in its uncompressed state, requires a large amount of information to describe its content. Therefore, recording or transmitting this information in its original form may be inefficient. Therefore, digital video needs to be compressed using various methods before recording or transmission. These compression methods include lossy coding and lossless coding. Lossy coding achieves high compression performance by sacrificing some image quality, while lossless coding prevents image quality degradation by sacrificing some compression performance. Regardless of the coding method used, to meet the need for obtaining high-quality digital video within limited storage recording capacity and communication transmission bandwidth, it is necessary to implement a technology that can achieve a high compression rate while minimizing the sacrifice of image quality.
[0009] As described above, the encoding process for compression requires various operations, such as: spatial segmentation of digital video, segmentation and / or processing of color channels, removal of spatial redundancy, removal of temporal redundancy, tracking of motion vectors within the video, encoding of differential images, quantization, coefficient scan, run-length coding, entropy coding, and a series of processes including loop filtering. These encoding operations typically consume computational resources and require a certain amount of time to complete. Similarly, the decoding operations for the above encoding operations also require certain computational resources and a certain amount of time. The main goal of video encoding and decoding technologies is to ensure that the aforementioned resource consumption and time consumption do not affect the production, recording, supply, and viewing of digital video.
[0010] Therefore, the present invention addresses the above-mentioned technical problems in the field of video encoding and decoding by providing a new technology that can play a role in at least one of the following aspects: improving encoding efficiency, improving decoding efficiency, improving video quality, reducing computational load, reducing software size, reducing hardware size, and improving other performance related to encoding and decoding.
[0011] Solution to the Problem
[0012] To solve the above-mentioned technical problem, a video decoding method based on geometric partitioning prediction according to an embodiment of the present invention includes: a step of reading geometric partitioning mode (GPM) information about the geometric partitioning of blocks from an encoded bitstream; a step of determining a first partitioning region and a second partitioning region based on the GPM information; a step of reading first prediction information from an encoded bitstream; a step of reading second prediction information from an encoded bitstream; and a step of generating a first prediction block about the first partitioning region based on the first prediction information, generating a second prediction block about the second partitioning region based on the second prediction information, and combining the first prediction block and the second prediction block to generate a final prediction block, wherein the first prediction information and the second prediction information are different from each other.
[0013] The step of generating the final prediction block may include: determining a boundary region including the boundary lines of the first segmentation region and the second segmentation region based on the GPM information; generating the boundary region prediction block; generating the final prediction block by combining the first prediction block, the second prediction block, and the boundary region prediction block; and storing boundary region information about the boundary region prediction block.
[0014] The boundary region prediction block may be generated based on the average of the weighted values of the first prediction block and the second prediction block.
[0015] The weighted value may be determined based on at least one of the following: the first prediction information, the second prediction information, the prediction information of the surrounding block, whether the prediction information of the surrounding block is intra-frame prediction information, the distance between the surrounding block and the generated boundary region, and the reference vector information of the surrounding block.
[0016] The boundary region information may be stored for reference when decoding other blocks subsequently.
[0017] The first prediction information may include first reference vector information, the second prediction information does not include reference vector information, and the boundary region information may be generated based on the first reference vector.
[0018] The second prediction information described above may be assigned at least one of the following: zero, an arbitrary random number, or a value indicating missing information, for at least one of the reference vector information and the reference image index.
[0019] The first prediction information regarding the first segmented region may include the first reference vector information, and the second prediction information regarding the second segmented region may include the second reference vector information. The boundary region information may be generated based on at least one of the first prediction information and the second prediction information described above.
[0020] The first prediction information may include at least one of the following: whether an intra-frame mode is used, whether a merged mode or an inter-frame mode is used, whether an intra-block copy (IBC) mode is used, and the type of intra-frame mode.
[0021] The first prediction information is about the intra-block copy (IBC) mode, the first reference vector represents a block vector, and the boundary region information is generated based on the first prediction information.
[0022] The first prediction information may include a first index of the first IBC reference vector candidate list.
[0023] The second prediction information may include at least one of the following: whether an intra-frame mode is used, whether a merged mode or an inter-frame mode is used, whether an intra-block copy (IBC) mode is used, and the type of intra-frame mode.
[0024] The second prediction information may be about the IBC mode, the second reference vector may represent a block vector, and the boundary region information may be generated based on the first prediction information and the second prediction information.
[0025] The first prediction information may include a first index of a first IBC reference vector candidate list, the second prediction information may include a second index of a second IBC reference vector candidate list, and the boundary region information may be generated based on third index information, wherein the third index information may be determined based on at least one of the first index and the second index.
[0026] The first prediction information may be related to inter-frame mode, the first reference vector may represent a motion vector, and the boundary region information may be generated based on the first prediction information.
[0027] The first prediction information may include a first index based on a first reference vector candidate list under merge mode.
[0028] The second prediction information may be related to inter-frame mode, the second reference vector may represent a motion vector, and the boundary region information may be generated based on the first prediction information and the second prediction information.
[0029] The first prediction information includes a first index based on a first reference vector candidate list under the merging mode, the second prediction information may include a second index based on a second reference vector candidate list under the merging mode, and the boundary region information may be generated based on third index information, wherein the third index information may be determined based on at least one of the first index and the second index.
[0030] The first prediction information may include bidirectional reference vectors, the second prediction information may include unidirectional reference vectors, and the boundary region information may be generated based on the first prediction information.
[0031] At least one of the above-mentioned first reference vector candidate list and the above-mentioned second reference vector candidate list can be configured to preferentially include at least one of the modes based on unidirectional reference vector information and the modes based on bidirectional reference vector information.
[0032] The above-mentioned boundary region information can be generated based on the reference vector obtained by weighted averaging of the above-mentioned first reference vector and the above-mentioned second reference vector.
[0033] To solve the above-mentioned technical problems, a video coding method based on geometric segmentation prediction according to an embodiment of the present invention includes: a step of determining a first segmentation region and a second segmentation region based on the geometric segmentation information of a block; a step of determining first prediction information about the first segmentation region; a step of determining second prediction information about the second segmentation region; and a step of encoding the above-mentioned first prediction information, the above-mentioned second prediction information and the above-mentioned combined information into a bitstream, wherein the above-mentioned first prediction information and the above-mentioned second prediction information are different from each other.
[0034] The above-mentioned method may include: a step of setting a boundary region including the boundary line of the above-mentioned first segmentation region and the above-mentioned second segmentation region; and a step of storing boundary region information about the above-mentioned boundary region prediction block.
[0035] The above-mentioned boundary region information can be stored for reference when encoding other blocks subsequently.
[0036] The above method may further include: encoding the difference between the first predicted block predicted based on the first prediction information and the actual image value of the region described in the first segmentation specification (page 3 / 22, CN 121488465 A), and the difference between the second predicted block predicted based on the second prediction information and the actual image value of the second segmented region as a residual into the bitstream.
[0037] The first prediction information may include first reference vector information, the second prediction information may not include reference vector information, and the boundary region information may be generated based on the first reference vector.
[0038] The second prediction information may be encoded as zero, an arbitrary random number, or a value indicating missing information for at least one of the reference vector information and the reference image index.
[0039] The first prediction information for the first segmented region may include first reference vector information, the second prediction information for the second segmented region may include second reference vector information, and the boundary region information may be generated based on at least one of the first prediction information and the second prediction information.
[0040] The aforementioned boundary region information can be generated based on the average of the weighted values of the first prediction block and the second prediction block.
[0041] The aforementioned weighted value can be determined based on at least one of the following: the aforementioned first prediction information, the aforementioned second prediction information, the prediction information of the surrounding blocks, whether the prediction information of the surrounding blocks is intra-frame prediction information, the distance between the surrounding blocks and the generated boundary region, and the reference vector information of the surrounding blocks.
[0042] The first prediction information may include at least one of the following: whether an intra-frame mode is used, whether a merge mode or an inter-frame mode is used, whether an intra-block copy (IBC) mode is used, and the type of intra-frame mode.
[0043] The first prediction information may be related to the IBC mode, the first reference vector may represent a block vector, and the boundary region information may be generated based on the first prediction information.
[0044] The first prediction information may include a first index of a first IBC reference vector candidate list.
[0045] The second prediction information may include at least one of the following: whether an intra-frame mode is used, whether a merge mode or an inter-frame mode is used, whether an intra-block copy mode is used, and the type of intra-frame mode.
[0046] The second prediction information may be related to the IBC mode, the second reference vector may represent a block vector, and the boundary region information may be generated based on the first prediction information and the second prediction information.
[0047] The first prediction information may include a first index of a first IBC reference vector candidate list, the second prediction information may include a second index of a second IBC reference vector candidate list, and the boundary region information may be generated based on third index information, wherein the third index information may be determined based on at least one of the first index and the second index.
[0048] The first prediction information may be related to inter-frame mode, the first reference vector may represent a motion vector, and the boundary region information may be generated based on the first prediction information.
[0049] The first prediction information may include a first index of a first reference vector candidate list based on a merged mode.
[0050] The second prediction information may be related to inter-frame mode, the second reference vector may represent a motion vector, and the boundary region information may be generated based on the first prediction information and the second prediction information.
[0051] The first prediction information may include a first index of a first reference vector candidate list based on a merged mode, the second prediction information may include a second index of a second reference vector candidate list based on a merged mode, and the boundary region information may be generated based on third index information, wherein the third index information may be determined based on at least one of the first index and the second index.
[0052] The first prediction information may include a bidirectional reference vector, the second prediction information may include a unidirectional reference vector, and the boundary region information may be generated based on the first prediction information.
[0053] At least one of the first and second reference vector candidate lists is preferably a mode based on unidirectional reference vector information and a mode based on bidirectional reference vector information.
[0054] The aforementioned boundary region information can be generated based on a reference vector obtained by weighted averaging of the aforementioned first reference vector and the aforementioned second reference vector.
[0055] The aforementioned GPM information may include constraint information determining at least one of the aforementioned first prediction information and the aforementioned second prediction information.
[0056] The aforementioned constraint information may include information specifying that at least one of the aforementioned first prediction information and the aforementioned second prediction information is any one of a merging mode, an inter-frame mode, an intra-block copy (IBC) mode, and an intra-frame mode.
[0057] The aforementioned constraint information may include information specifying that at least one of the aforementioned first prediction information and the aforementioned second prediction information does not include any one of unidirectional motion information and bidirectional motion information.
[0058] The aforementioned constraint information may include information specifying that prediction modes other than the aforementioned geometric segmentation-based predictions must not be combined.
[0059] To solve the above-mentioned technical problems, a video decoder device based on geometric segmentation prediction according to an embodiment of the present invention includes: a bitstream parsing unit, configured to read geometric segmentation pattern (GPM) information, first prediction information and second prediction information about the geometric segmentation of blocks from an encoded bitstream; a geometric segmentation prediction unit, configured to determine a first segmentation region and a second segmentation region based on the GPM information, generate a first prediction block about the first segmentation region based on the first prediction information, generate a second prediction block about the second segmentation region based on the second prediction information, and generate a final prediction block by combining the first prediction block and the second prediction block; and a memory, configured to store decoded image information including the final prediction block, wherein the first prediction information and the second prediction information are different from each other.
[0060] To solve the above-mentioned technical problems, a video encoder device based on geometric segmentation prediction according to an embodiment of the present invention includes: a block processing unit for receiving block data to be encoded; a geometric segmentation unit for determining a first segmentation region and a second segmentation region based on the geometric segmentation information of the block, determining first prediction information about the first segmentation region, and determining second prediction information about the second segmentation region; and an encoding unit for encoding the first prediction information and the second prediction information into a bitstream, wherein the first prediction information and the second prediction information are different from each other.
[0061] Effects of the Invention
[0062] According to the present invention, reference vector information can be stored more effectively for boundary region blocks using prediction value weighted averaging.
[0063] According to the present invention, it is possible to store reference vector information for cases using intra-frame modes that do not include reference vector information. Brief Description of the Drawings
[0064] FIG1 is a conceptual diagram of a video communication system according to an embodiment of the present invention.
[0065] FIG2 is a conceptual diagram of the arrangement of encoders and decoders in a real-time video stream environment according to an embodiment of the present invention.
[0066] FIG3 is a conceptual diagram of the functional unit of a video decoder according to an embodiment of the present invention.
[0067] FIG4 is a conceptual diagram of the functional unit of a video encoder according to an embodiment of the present invention.
[0068] FIG5 is a conceptual diagram of a frame type according to an embodiment of the present invention.
[0069] FIG6 is a conceptual diagram showing the structure of a video encoder using VVC.
[0070] FIG7 is a conceptual diagram showing a geometric segmentation prediction mode.
[0071] FIG8 is a conceptual diagram showing the problem of storing reference vector information for segmented regions. Specification 5 / 22 pages 9 CN 121488465 A
[0072] FIG10 is a conceptual diagram showing bidirectional motion vector and index-priority storage according to an embodiment of the present invention.
[0073] FIG11 is a conceptual diagram showing the priority addition of unidirectional reference vector information to the merging mode list according to an embodiment of the present invention. Detailed Description
[0074] The present invention can be modified in various ways and has various embodiments, and specific embodiments are illustrated in the accompanying drawings and described in detail. However, this is not intended to limit the invention to a specific implementation, but should be understood as including all modifications, equivalents, and even substitutions that fall within the scope of the ideas and techniques of the invention.
[0075] Although terms such as “first” and “second” may be used to describe various components, the aforementioned components should not be limited by the aforementioned terms. The purpose of the aforementioned terms is to distinguish one component from another. For example, without departing from the scope of the invention, a first component may be named a second component, and similarly, a second component may be named a first component. The term “and / or” includes a combination of multiple related described items or one of multiple related described items, and is not exclusive unless otherwise stated. The items listed in this application are merely exemplary descriptions intended to facilitate understanding of the concepts and possible implementations of the invention, and are therefore not intended to limit the scope of the embodiments of the invention.
[0076] “A or B” in this specification may refer to “A only”, “B only”, or “A and B”. In other words, “A or B” in this specification may also be interpreted as “A and / or B”. For example, “A, B or C” in this specification may mean “A only”, “B only”, “C only”, or “any combination of A, B and C”.
[0077] The forward slash ( / ) or comma used in this specification may mean “and / or”. For example, “A / B” may mean “A and / or B”. Accordingly, “A / B” may mean “A only”, “B only”, or “A and B”. For example, “A, B, C” may mean “A, B or C”.
[0078] The phrase “at least one of A and B” in this specification may refer to “only A”, “only B”, or “A and B”. Furthermore, the expressions “at least one of A or B” or “at least one of A and / or B” in this specification may be interpreted equivalently as “at least one of A and B”.
[0079] Additionally, the phrase “at least one of A, B and C” in this specification may refer to “only A”, “only B”, “only C”, or “any combination of all A, B and C”. Furthermore, “at least one of A, B or C” or “at least one of A, B and / or C” may refer to “at least one of A, B and C”.
[0080] When a component is described as “connected” or “accessed” to another component, it should be understood that it may be directly connected to or accessed to another component, or there may be other components in between. Conversely, when a component is described as “directly connected” or “directly accessed” to another component, it should be understood that there are no other components in between.
[0081] The terminology used in this application is merely illustrative of particular embodiments and not limiting of the invention. Where there is no obvious singular expression in the context, the singular description includes the meaning of the plural. In this application, terms such as “comprising” or “possessing” indicate the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, rather than precluding the presence or additional possibility of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.
[0082] Unless otherwise specified, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms that are generally used the same as those defined in a dictionary have the same meaning as in the context of the related art and, unless explicitly defined, do not have an ideal or excessive meaning in this application. Specification page 6 / 22 10 CN 121488465 A
[0083] When describing the present invention in this application, embodiments may be described or illustrated in the form of unit blocks that perform the described functions or functions. In this application, the above-mentioned blocks may represent one or more devices, units, modules, components, etc.The aforementioned blocks can be implemented in hardware using one or more logic gates, integrated circuits, processors, controllers, memories, electronic components, or information processing hardware methods including but not limited to these. Alternatively, the aforementioned blocks can be implemented in software using application software, operating system software, firmware, or information processing software methods including but not limited to these. A block can be divided into multiple blocks that perform the same function, and conversely, the function of multiple blocks can also be performed by one block. The aforementioned blocks can also be implemented as physically separated or combined according to any standard. The aforementioned blocks can also be implemented to operate in an environment where their physical location is not specified and they are isolated from each other through communication networks, the Internet, cloud services, or communication methods including but not limited to these. All of the above implementation methods fall within the scope of various embodiments that can be adopted by those skilled in the art of information and communication technology to achieve the same technical idea. Therefore, any specific implementation method should be interpreted as falling within the scope of the inventive technical idea of this application.
[0084] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. To facilitate an overall understanding of the present invention, the same components in the drawings will be represented by the same reference numerals, and repeated descriptions of the same components will be omitted. Furthermore, the multiple embodiments are not mutually exclusive, and some embodiments can be combined with one or more other embodiments to form new embodiments.
[0085] Digital Video Codec
[0086] FIG1 is a conceptual diagram of a video communication system according to an embodiment of the present invention. The video communication system 100 described above can be configured to include at least two terminals 110 and 120 interconnected via a network 105.
[0087] In one embodiment of the present invention, FIG1 may represent a block diagram for configuring a unidirectional video communication network. In order to send video data 111 via the network 105, the first terminal 110 of the terminals may encode the video data. The second terminal 120 of the terminals may be configured to receive the encoded video data 121 via the network and decode and display it.
[0088] In another embodiment of the present invention, FIG1 may represent a block diagram for configuring a bidirectional video communication network. In order to realize the bidirectional video communication described above, each of the terminals 110 and 120 may be configured to encode the video data it acquires in order to transmit video 112 and 122 to other terminals via the network. Each terminal can also be configured to receive video data sent by other terminals 113 and 123 via the network for decoding and display the decoded video data.
[0089] The terminals 110 and 120 shown in FIG1 can be, according to the embodiment, devices such as server computers, personal computers, portable computers and smartphones, but are not limited thereto; any commonly used computing device can be used.For example, according to embodiments of the present invention, the aforementioned terminals 110, 120 may refer to desktop computers, laptop computers, tablet PCs, mobile phones, smartphones, personal digital assistants (PDAs), workstations, electronic calculators, server computers, cloud computers, virtual computers, quantum computers, or any other electronic, electrical, or quantum computing device implemented in a portable or non-portable form. In particular, the aforementioned devices may also be interpreted as any device designed to operate as a terminal device according to an embodiment of the present invention, or capable of operating as a terminal device according to an embodiment of the present invention, or capable of operating as a terminal device according to an embodiment of the present invention, or capable of installing and / or running computer programs to perform methods corresponding to the operation.
[0090] The aforementioned terminals 110, 120 may be implemented by multiple functional units that are interconnected in various forms, such as buses, circuits, or relationships between routines and subroutines, and configured to exchange information within the aforementioned terminals 110, 120. Furthermore, through the aforementioned interconnections, these terminals can be configured to include a processor 130 with computing capabilities and a memory 140 connected to the processor, to execute or support the operation of functional units that primarily require calculations between the aforementioned functional units.
[0091] The processor 130 described in this specification may refer to one or more general-purpose or special-purpose computers, such as processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field-programmable gate arrays (FPAs), programmable logic units (PLUs), microprocessors, or any other means capable of executing and responding to instructions.
[0092] For ease of understanding, even though the processor 130 is referred to only in the singular, those skilled in the art should understand that the processor 130 may include multiple processing elements and / or various types of processing elements. For example, according to one embodiment of the present invention, the apparatus may include a plurality of processors, or a processor and a controller, as described above as processor 1 3 0.Furthermore, the processor 130 described above can be implemented using various processing configurations, such as a parallel processor or a multi-core processor.
[0093] The processor 130 described above can be configured to execute an operating system (OS) and one or more software programs running on the operating system. Furthermore, the processor can access, store, manipulate, process, and generate data in response to the execution of the software. The software may include a computer program, code, instructions, or a combination thereof, and can be configured as needed, or can provide independent or collective instruction control over the processing device. The software may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, so that it can be parsed by the processor 130, or provide instructions or data to the processor. The aforementioned software can be distributed among multiple computer systems (e.g., terminals 110 and 120) connected to the aforementioned network 105, and stored or executed in a distributed manner.
[0094] The aforementioned software can also be implemented in the form of program instructions executable by various computer means, and recorded or stored in the aforementioned memory 140. The aforementioned memory 140 can be a computer-readable recording medium, in which program commands, data files, and data structures can be recorded individually or in combination. The program commands stored in the aforementioned memory 140 can be implemented based on a command system specifically designed and configured for embodiments of the present invention, or can follow a command system known and available to those skilled in the art of computer software, such as the command systems of assembly language, C, C++, Java, Python, etc. It should be understood that the aforementioned command system and the program commands it generates include not only machine language code generated by a compiler, but also high-level language code that can be executed by an interpreter or other means by a device of an embodiment of the present invention and / or the aforementioned processor 130.
[0095] A computer-readable recording medium constituting an embodiment of the apparatus of the present invention includes the memory 140 described herein, and may include: temporary or volatile recording media such as processor cache, random access memory (RAM), flash memory, etc., which are maintained only during the operation of the processor; or relatively non-volatile or long-term recordable recording media such as hard disk, floppy disk, magnetic tape, etc.; optical recording media such as CD-ROM, DVD, etc.; magneto-optical media such as floppy disk; or solid state memory; or read-only recording media such as a hardware-mounted read-only memory (ROM). Furthermore, since the circuit wiring is configured to execute a series of program instructions and equivalent actions through a hard-wired structure, it can be considered that the various steps involved in executing the actions described in the embodiments of the present invention can be regarded as being recorded through the connection and arrangement of the hardware components. Therefore, it is obvious to those skilled in the art that this connection and arrangement method can be regarded as equivalent to the memory 140 described above. Specification 8 / 22 pages 12 CN 121488465 A
[0096] The above embodiments regarding the processor 130 and the memory 140 are not exclusive and can be selected or combined as needed. For example, a hardware device can be configured to perform the actions of an embodiment of the present invention as a module composed of more than one of the above software, and vice versa. As another example, in this specification, all or part of the actions assigned to a certain functional unit can be implemented by more than one of the above software stored in a device according to an embodiment of the present invention (preferably stored in a recording medium belonging to the scope of the above memory) and configured to be executed by the processor. In this case, such a functional unit can be referred to as a functional unit "included in" the processor.
[0097] The present invention is applicable to any environment for establishing a one-way or two-way video communication network, and it should be understood that the network 105 can be configured to transmit encoded video data between the terminals 110 and 120 by any means.
[0098] In one embodiment of the present invention, the network 105 may refer to a wired or wireless communication network. In this embodiment, the network can be configured to communicate information using any communication standard, including packet-based communication. Packet communication can be understood to include data packets such as TCP or UDP.The wired communication method of the aforementioned network 105 can be a method of connecting to an external communication network via RJ-11 standard telephone lines, various types of Ethernet cables conforming to RJ-45 standard, other coaxial cables, metal cables, optical cables and other various wired media. According to the embodiments, the wireless communication method of the network 105 described above may include short-range wireless communication methods, such as Bluetooth, Wi-Fi, Zigbee, and near field communication (NFC); or may include long-range wireless communication methods, such as Wibro, WiMax, Global Systems for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long-Term Evolution (LTE), New Radio (NR), and other wireless communication technology names collectively referred to as 2G, 3G, 4G, 5G, and 6G communication standard generations, as well as international standard wireless communication standards such as IMT-2000, IMT-Advanced, IMT-2020, and IMT-2030. Of course, apart from this, any existing or newly developed wired or wireless communication means, methods, standards, and protocols applied to the implementation of the network 105, as long as they are configured to transmit and receive in the information communication devices such as the terminals 110 and 120, will not affect the achievement of the purpose of this invention. Furthermore, it is obvious that a network 105 can be configured using a combination of more than one wired and / or wireless standard.
[0099] However, in another embodiment of the invention, the network 105 may also be understood to include a process of transmitting information using a computer-readable recording medium. In this case, the configuration of the network is not limited to the communication medium, but should be understood to include a process of temporary storage and physical transmission in computer-readable storage and / or recording media. Computer-readable recording media used for the aforementioned information transmission can be understood as relatively non-volatile or long-term recording media, which are mainly used as a means of transmitting data between computing devices, especially magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floppy disks; or solid-state memory.
[0100] Furthermore, regardless of the information communication or transmission means used, as long as it supports the transmission and decoding of video data in an encoded state, it can be considered to fall within the scope of the embodiments of the present invention. Therefore, except for some of the examples listed above, all known or future information communication or transmission means can fall within the scope of application of the present invention.
[0101] FIG2 is a conceptual diagram of the arrangement of encoders and decoders in a real-time video streaming environment according to an embodiment of the present invention. The streaming media system 200 shown in FIG2 can be applied to video data communication networks, including, for example, digital broadcasting, video telephony, and video conferencing. However, even when it involves information transmission through a recording medium, the same or similar technical structures of the above streaming media system can be applied as described above.
[0102] According to an embodiment of the present invention, the above streaming media system may include a video source 210 that generates the video stream. The video source may include a digital video acquisition unit 212 for acquiring uncompressed raw video, for example, the digital video acquisition unit may be configured as a digital camera or other device. The original video stream 215 may have a large capacity, and therefore can be compressed by combining with or connected to the video encoder 217 of the video source.
[0103] The encoder 217 may be configured as a unit including hardware, software or a combination of both, for implementing a video encoding method and / or its implementation according to an embodiment of the present invention.
[0104] Through the encoder 217, an encoded bitstream 219 with a reduced capacity compared to the original video stream can be output. The bitstream 219 may be provided in real time for communication via a relay device (e.g., a streaming media server 220), and / or may be stored on the recording medium 225 of the streaming media server 220 for subsequent use.
[0105] The streaming media system 200 may include at least one streaming media client 230, 240 that accesses the streaming media server 220 to receive or later acquire the encoded bitstream 229. The aforementioned streaming media client may include a video decoder 232, which acquires the encoded bitstream 229 (which can also be considered a copy of the bitstream 219 received by the aforementioned streaming media server), decodes the bitstream 229, and outputs the resulting video data as video data that can be displayed on a display 235 or other visual, auditory, or other sensory display means.
[0106] As described above, the encoding and decoding functions of video data are collectively referred to as a coder-decoder system, i.e., a video codec.
[0107] FIG3 is a conceptual diagram of the functional units of a video decoder according to an embodiment of the present invention.As shown in FIG3, the receiving unit 310 can receive at least one encoded video data to be decoded by the decoder 305. In one embodiment of the present invention, each received encoded video data can be independent, and the decoding process of each independent video data can be independent of the decoding process of other video data. The encoded video data can be received to the receiving unit 310 through a hardware or software connection 315 between the receiving unit and the storage device. As mentioned above, the storage device can refer to a streaming media server located at the other end of the communication network, or it can be a physical storage medium, but is not limited thereto.
[0108] The receiving unit 310 can receive the encoded video data and other accompanying data together, such as encoded audio data or other auxiliary data, and each of the above data can be separated from the video data and provided to other appropriate processing units 312 other than the video decoder.
[0109] When the video data is received through the communication network, a buffer memory 320 can be connected between the receiving unit 310 and the decoder 305 to minimize the latency and interruption caused by the network environment. The aforementioned buffer memory 320 may be a computer-readable recording medium used to temporarily store the received video data and stably provide it to the parser 330 corresponding to the input of the decoder 305. However, if the bandwidth of the communication network is sufficient, or the video data is read from a local and non-physically isolated recording medium, or if no communication delay is expected in other environments, the aforementioned buffer memory may not be necessary.
[0110] The video decoder 305 may include the aforementioned parser 330 as its input for parsing the aforementioned encoded video data. The parser may separate (parsing) various information stored in the bitstream form of the aforementioned encoded video data according to predetermined rules, and perform entropy decoding 335 on the entropy-coded video data when needed, thereby performing the function of reconstructing the symbols 338 as segments of video encoded information. The aforementioned symbols 338 may include any information for controlling the operation of the decoder 305, and / or may further include information for controlling the devices (such as display devices) attached to the decoder 305 that perform the operation. Control information for controlling the aforementioned display device may include information in a format called supplementary enhancement information (SEI) or video usability information (VUI).
[0111] As described above, the parser 330 may be configured to perform entropy decoding 335 on the aforementioned encoded video data.The entropy coding method for the encoded video data described above (page 10 / 22, CN 121488465 A) may vary depending on the aforementioned coding standard, and the decoding operation should be performed accordingly. Representative examples of the aforementioned entropy coding standards may include variable length coding, Huffman coding, and arithmetic coding. The various coding methods described above may be context-adaptive or context-sensitive methods according to the standard, and may also be based on principles well known to those skilled in the art.
[0112] The parser 330 described above may be configured to extract at least one local image from the aforementioned encoded video data. The definition of the aforementioned local image may vary depending on the aforementioned coding standard, and according to the standard, one of the examples listed below may apply, or multiple examples may be applied simultaneously. For example, the aforementioned local image can be defined as the following units: group of pictures (GOPs), pictures / frames, tiles, slices, macroblocks, blocks, subblocks, transform units (TUs), prediction units (PUs), etc.
[0113] The aforementioned parser 330 can be configured to extract encoded information from the aforementioned encoded video data, such as transform coefficients, quantization parameters (QPs), and / or motion vectors. The aforementioned parser 330 can be configured to perform entropy decoding 335 and parsing actions on the video data received from the aforementioned buffer memory, and selectively decode the symbols 338 representing the aforementioned encoded information. Furthermore, the parser 330 can also be configured to selectively provide specific symbols 338 to specific decoding function units within the decoder 305, such as inverse quantization and inverse transform units 340, intra prediction units 350, inter prediction units 355, or loop filter units 360, etc.The control provided by such information can be determined by the arrangement of information contained in the encoded video, and may vary depending on the encoding standard. It is not limited to the scope of the embodiments of the present invention and is not described in detail in this conceptual diagram.
[0114] The decoder 305 described above may be configured with a plurality of conceptual functional units for receiving and processing the encoded information from the parser 330 described above. It should be understood that these conceptual functional units may be combined or further subdivided according to implementation needs. For example, they may be further separated for ease of implementation; they may be integrated together for improved operating efficiency. In either case, the functional units may be configured to interact closely with each other. Despite the possibility of such integration or separation, the following description will still be based on the combination of conceptual functional units in order to clarify the decoding process of video data applied as an embodiment of the present invention.
[0115] The decoder described above may include an inverse quantization and inverse transform unit 340. The aforementioned inverse quantization and inverse transform unit 340 can be configured to receive encoded information from the aforementioned parser 330, the encoded information including a method for numerical transformation, block size, quantization coefficients for recovering quantized information, and distinguishing information for simplifying and representing the quantization coefficients, and can be configured to output block values 341 that can be input to the aggregator 370 as a result of processing the aforementioned encoded information.
[0116] In one embodiment of the present invention, the output value of the aforementioned inverse quantization and inverse transform unit 340 may include intra-frame predicted encoded block values. The aforementioned intra-frame predicted block values refer to values that can be decoded by using the prediction information of the currently being decoded local image (e.g., the current intra-frame) without using previously decoded local images (e.g., prediction information from the previous frame).
[0117] The prediction information in the current local image may be provided by the intra-frame prediction unit 350. According to an embodiment of the present invention, the intra-frame prediction unit 350 uses image information of spatially adjacent regions extracted from a local image that is currently being decoded and has been partially decoded to generate block values as prediction information that are in the same block form as the block being decoded. The local image information can be provided 381 by the current image buffer (i.e., the so-called line buffer 380). According to an embodiment, the merging unit 370 can be configured, as described on page 11 / 22 of CN 121488465 A, to merge the prediction information 351 generated by the intra-frame prediction unit 350 with the block values 341 provided by the inverse quantization and inverse transform unit 340.
[0118] In another embodiment, the output value of the inverse quantization and inverse transform unit 340 can be the block values encoded by inter-frame prediction, and in some cases, the output value can include block values that have undergone motion compensation.In this case, the inter-frame prediction unit 355 can extract and use sample information 386 for motion-based prediction from the reference picture buffer 385. The information 356 obtained by performing motion compensation on the sample information based on the symbols 338 contained in the block value as the output value can be configured to be merged by the merging unit 370 with the block value 341 provided by the inverse quantization and inverse transform unit 340. In this case, the block value 341 can be referred to as the differential value or residual value.
[0119] The position information in the memory, used by the inter-frame prediction unit 355 to extract the sample information from the reference image, can be determined, for example, by combining X, Y and other symbols 338 used to indicate a specific position in the reference image and by the motion vector transmitted to the inter-frame prediction unit 355. The inter-frame prediction unit 355 may also include the following functions: when providing motion vectors that support "subsampling", it can process the sample values by using interpolation; and it can predict and enhance the motion vector values.
[0120] The output value 371 of the merging unit 370 may be provided to the loop filter unit 360 and processed by various loop filtering methods. The loop filter unit 360 may also be configured to receive not only the block unit output 371 of the merging unit 370, but also the symbol 338 from the parser 330 to control its operation. The output of the loop filter unit 360 may be transmitted to an external display device such as the display device described above via the output connection 390. However, in order to be used for prediction and parsing of subsequent intra-frame or inter-frame coded block values, 361 may be stored in the line buffer 380 and then stored in the reference image buffer 385.
[0121] After decoding, a specific local image (e.g., a frame) may be used as a reference image for performing predictive decoding in subsequent decoding processes. A local image (e.g., a frame) can be progressively accumulated into line buffer 380 and decoded. Once a frame is decoded, the contents of line buffer 380 are transferred (383) to reference image buffer 385, and a new line buffer 380 is allocated for decoding the new frame.
[0122] The video decoder 305 can be configured to perform decoding operations according to a predetermined video compression technique, which can be documented based on various international standards or commercial standards. These standards may include, for example, international standard recommendations such as H.264, H.265, and H.266 defined by the International Telecommunication Union Standardization Sector (ITU-T).Those skilled in the art should understand that the above recommendations are equivalent to international standards jointly developed by the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC). Encoded video data may conform to a specific bitstream syntax defined by the relevant standards, which is explicitly specified in video compression specification documents and standard documents, and is specifically defined and constrained by the profiles and levels specified therein. To ensure compliance with the above profiles and levels, the complexity of the encoded video data also needs to be limited to a certain level. For example, certain profiles or levels may configure limitations such as maximum image size, maximum decoding rate, and maximum reference image size. In some embodiments, the above limitations may be further restricted by the hypothetical reference decoder (HRD) and the HRD buffer management metadata signals included in the encoded video data.
[0123] According to one embodiment of the present invention, the receiving unit 310 may receive additional redundant data along with the encoded video. The additional data may be considered as part of the encoded video data. The aforementioned additional data may include information for use by the decoder 305 to correctly decode the data or more accurately reconstruct an image close to the image before encoding (page 12 / 22 of the specification, CN 121488465 A). The aforementioned additional data may be provided in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, and forward error correction codes.
[0124] FIG4 is a conceptual diagram of the functional units of a video encoder according to an embodiment of the present invention. The encoder 405 may be configured to receive raw video information 402 from a video source 401 and perform encoding.
[0125] The raw video information 402 may have any suitable bit depth, such as 8 bits, 10 bits, 12 bits, etc. Furthermore, the raw video information 402 may have any suitable color space, such as R / G / B, Y / U / V, Y / Cb / Cr, etc. Furthermore, the aforementioned original video source can have any suitable sampling structure corresponding to the aforementioned color space, such as Y / Cb / Cr 4:2:0, Y / Cb / Cr 4:4:4, etc. The original video source having this predetermined format can be provided to the aforementioned encoder in the form of a digital video stream.
[0126] In a one-way video communication network, the aforementioned original video information 402 can be obtained from a recording medium storing a pre-prepared original video clip. In a two-way video communication network, the aforementioned original video information 402 can be obtained from an image acquisition device (e.g., a camera) included in the two-way video communication that generates at least one video transmission stream.
[0127] The video data including the original video information 402 described above can be configured to play in chronological order as multiple partial images to simulate motion. These partial images can be represented using concepts such as pictures or frames. Depending on the sampling structure, color space, etc., used, the partial images may include more than one sample. Those skilled in the art will understand that the term "sample" is closely related to "pixel" in digital images. The working principle of the encoder will be explained below in relation to such samples.
[0128] According to one embodiment of the present invention, the encoder 405 can be configured to encode and compress the (partial) images constituting the original video information 402 in real time (or according to other time requirements required by the implementation method) into the form of encoded video information.
[0129] In the encoder 405 described above, the control unit 450 can be a functional unit for appropriately controlling the encoding speed. As described below, the control unit 450 can be configured to control other functional units and to perform functional combinations with the functional units described below. The parameters set by the control unit 450 may include parameters related to bitrate control, such as image skipping, quantizer, and variable values for applying image quality optimization techniques. Additionally, they may include values such as image size, group of pictures (GOP) structure, and maximum search range of motion vectors. Those skilled in the art will understand that the control unit 450 may have various other functions, which can be added or removed depending on the video encoder design optimized for individual system designs.
[0130] According to one embodiment of the present invention, the encoder 405 may be configured to operate in a structure known to those skilled in the art, such as a "coding loop." For simplicity, exemplarily, the encoding loop may consist of an internal encoder (also called a "source encoder") 410 and an internal decoder 420, wherein the internal encoder is responsible for receiving the image to be encoded and generating symbols based on at least one previously encoded reference image, and the internal decoder is configured to connect to the internal encoder. The aforementioned internal decoder 420 can be configured to receive the output of the aforementioned internal encoder 410 and perform a reproduction operation of the sample data generated after the remote decoder 490, which is actually located remotely, receives the encoded video information from the aforementioned encoder 405.
[0131] The video data composed of the sample data reconstructed by the aforementioned internal decoder 420 can be configured to be input into the reference image buffer of the aforementioned encoder 405.As described above, since the internal decoder 420 is implemented to reproduce the output of the encoder 405 and the result of decoding by the remote decoder, the video data recorded in the reference image buffer can be identical in bit units to the information in the reference image buffer of the remote decoder. That is, the prediction function unit included in the encoder 405 can read from the reference image buffer of the encoder 405 the same value as the previous frame sample value that the subsequent decoder will reference during the decoding process.
[0132] As described above, the principle of achieving consistency of the reference image buffer between the encoder 405 and the decoder 490 through the internal decoder 420 at the encoder 405 end is well known to those skilled in the art. In addition, those skilled in the art are also well aware of how to deal with situations where such a guarantee cannot be made (e.g., information loss due to communication failure).
[0133] An embodiment of the operation method of the internal decoder 420 has been described in detail with reference to FIG3. The decoder in FIG3 can be regarded as the "remote" decoder 490 described above. The aforementioned internal decoder 420 may not include lossless encoding and decoding parts, such as the parser 330 or entropy decoder 335. This is because the implementation of the aforementioned internal encoder 405 is only to reproduce the operation of the remote decoder, so it can directly decode symbols without performing symbol compression and decompression processes. Therefore, as shown in FIG3, the parser and entropy decoder may not have the previous functional units, or at least partially implement the previous functional units.
[0134] As described above, according to a preferred embodiment of the present invention, any decoder functional unit (excluding the parser and entropy decoder) present in the decoder can naturally exist in the corresponding encoder 405 as substantially the same functional unit.
[0135] The operation of the encoding functional unit that may be included in the aforementioned encoder 405 can be regarded as the inverse process of the aforementioned decoder functional unit. Therefore, this embodiment can usually be explained by reversing the operation of the aforementioned decoder functional unit. For example, quantization and transform functional units corresponding to inverse quantization and inverse transform units can be provided, as well as inter-frame prediction coding units corresponding to inter-frame prediction units can be provided. In addition, some additional explanations are provided below.
[0136] The internal encoder 410 described above may be configured to perform encoding for input image information (e.g., input frame) according to a predictive encoding method performed by a predictive encoding unit 440 that operates via a reference image buffer 430, for at least one reference image information, such as video data designated as a reference frame having at least one previously encoded local image (e.g., a pin) in time sequence.In this case, the encoder 405 can be configured to encode the difference between the sample blocks constituting the input image and the sample blocks constituting the reference image.
[0137] The internal decoder 420 can decode video data that can be designated as the reference image from the symbols generated by the internal encoder 410. As described above, since this video data is the same as the decoding action performed by the remote decoder, the video data used as the reference image can be provided to the encoder 405 in a lossy compressed and somewhat corrupted form, which is intended to ensure consistency with the decoder's action.
[0138] The prediction coding unit 440 can be configured to perform a prediction search action within the encoder 405. The prediction search action can refer to the inter-frame prediction or intra-frame prediction action described in the description of the decoder. For image information that has been input and is planned to be newly encoded, in order to obtain the motion vector, block shape, and metadata that may include these, as well as the sample blocks that are actually referenced, of the reference image's position information that can provide suitable prediction reference information for the new image information, the prediction unit can access the reference image buffer 430 and retrieve the information. The aforementioned predictive coding unit 440 can operate based on a so-called "sample block by pixel block" reference to obtain appropriate predictive reference information. According to one embodiment of the invention, as can be determined based on the search results obtained by the aforementioned predictive coding unit 440, at least one predictive reference information can be specified for the aforementioned input image to point to at least one reference image information stored in the aforementioned reference image buffer 430.
[0139] In one embodiment of the invention, the aforementioned control unit 450 can be configured to include setting parameters for video data encoding to manage the overall encoding operation of the internal encoder 410.
[0140] The outputs of all the aforementioned functional units can be processed by entropy coding 460 to achieve the final output. For the symbols generated by the aforementioned functional units, the aforementioned entropy coding 460 uses the various entropy coding techniques described above (which may include, for example: variable length coding, Huffman coding, and arithmetic coding) to encode them. Depending on the different standards, the above-mentioned encoding methods can be context-adaptive or context-sensitive methods. In addition, they can also be implemented based on principles well known to those skilled in the art.This entropy encoding 460 can typically achieve lossless compression, therefore, at least one symbol generated by the above-mentioned functional unit can be converted into encoded video data.
[0141] In the process of controlling the operation of the encoder 405, the control unit 450 can apply the encoding type belonging to a specific local image to each local image (e.g., a picture or a frame) within the encoding interval. The encoding method of the local image is also affected by the above-mentioned type. According to an embodiment, the above-mentioned type may include the "frame type" distinguished as follows.
[0142] FIG5 is a conceptual diagram of a frame type according to an embodiment of the present invention. The following description refers to FIG5.
[0143] An intra-frame ("I") image 510 may refer to an image that is encoded or decoded using only its own information without the need to predict and encode other picture information in the reference video data. According to the video coding standard, the above-mentioned "I" image may refer to a key frame, an independent / instantaneous decoder refresh (IDR) frame, a clean random-access (CRA) frame, etc. As described above, the “I” images referred to by various names can have various variations and applications within the limits allowed by various standards, and there can be some differences between them. In addition to the methods listed above, various application methods for implementing “I” images can be various methods known to those skilled in the art or newly provided.
[0144] The prediction (P) image 520 can refer to an image used to predict the sample values of the blocks constituting the above image, which can be at least one prediction information specifying at least one reference image, and / or an image encoded or decoded based on motion vectors and through intra-frame or inter-frame prediction. According to the video coding standard, the above “P” image can be configured to reference only one reference frame, or to reference more than one reference frame. When referencing more than one reference frame, sample information and / or related metadata derived from multiple reference images can be used to reconstruct a single block. However, generally, an image designated as a “P” image can be understood as limited to referencing and referencing images that are earlier in time.
[0145] The bidirectional prediction (B) image 530 may refer to at least one prediction information from at least two or more reference images specified for predicting sample values of blocks constituting the image, and / or an image encoded or decoded based on motion vectors and through intra-frame or inter-frame prediction. Typically, an image designated as a "B" image is distinct from an image designated as a "P" image and can be understood as an image that performs the reference, but is not limited to temporally previous images.
[0146] During the encoding and decoding process, video data can be spatially divided according to multiple sample blocks and encoded according to the block units described above. As is well known, the block units may include, for example, sizes in units of horizontal / vertical pixels such as 4x4, 8x8, 4x8, or 16x16, but are not limited thereto. The blocks may be encoded by a predictive coding method, based on their respective local images and, depending on the specified type (allowed and / or restricted), by referencing any other (already encoded) blocks. For example, the blocks of “I” image 510 may not use a predictive coding method, or may be encoded by referring to already encoded blocks in the same local image. That is, only the so-called intra-frame prediction method may be used. In contrast, “P” image 520 may further refer to at least one reference image encoded in a previous time unit, and therefore may also be encoded using inter-frame prediction along with intra-frame prediction. “B” image 530 may also refer to a reference image that was previously encoded in the encoding order but was executed later in the time unit. However, it is well known that even within a “P” or “B” frame, there may be blocks that are encoded without relying on predictive coding. However, it is well known that blocks that are encoded without using predictive coding may exist within a “P” or “B” image.
[0147] The video encoder 405 described above can be configured to perform encoding operations according to predetermined video compression techniques documented according to various international or commercial standards. Examples of the aforementioned standards may include all standards described in the decoder described above.
[0148] According to one embodiment of the invention, the transmitting unit 470 can buffer the encoded video data generated by the entropy coding described above (page 15 / 22 of specification 19 CN 121488465 A) so that the video data (ultimately transmitted to the remote decoder 490) can be provided / sent to the device storing the encoded video data via a hardware or software connection 495. According to an embodiment, when the transmitting unit 470 receives the encoded video data provided / sent from the video encoder 405, it can receive and merge other data (e.g., encoded audio data or other auxiliary data) accompanying the encoded video data from a separate source 480.
[0149] According to one embodiment of the present invention, the transmitting unit 470 may also be configured to transmit additional data along with the encoded video. The additional data may be considered as part of the encoded video data. The additional data may include information for the decoder, which is used to correctly decode the data or to more accurately reconstruct information closer to the pre-encoded image. Examples of the additional data may include all the examples described above with respect to the decoder receiving unit 310.
[0150] As described above, the present invention can be implemented using digital video compression standards widely used and understood by those skilled in the art.The aforementioned digital video compression standards may include at least one known compression standard, such as MPEG-2, MPEG-4 video, H.263, H.264 / AVC, H.265 / HEVC, H.266 / VVC, VC-1, AVI, QuickTime, VP-9, VP-10, Motion JPEG, etc.
[0151] Figure 6 is a conceptual diagram showing the structure of a video encoder using VVC. The structure shown in Figure 6 corresponds to well-known standard encodings such as ITU-T H.266 and ISO / IEC 23090-3, as well as the general structure of a video encoder called MPEG-I Part 3 or Versatile Video Coding (VVC).
[0152] According to Figure 6, the video encoder 605 may be configured to receive an encoded bitstream 602 as input and output raw video data 601 that is neither compressed nor encoded. During intra-frame encoding, the video data 601 can be directly provided to the luma mapping unit 610a, or it can be provided to the luma mapping unit 610b via the inter-frame prediction unit 620, which includes motion vector extraction. In the case of intra-frame encoding, the mapped luma signal can be provided not only to the output combiner 606 individually, but also to the output combiner 606 by selecting either the intra-frame prediction encoded signal output by the intra-frame prediction unit 625 (608) or the inter-frame prediction encoded signal output by the inter-frame prediction unit 620 from the luma mapping unit 610b. The result of the output combiner can be applied to the chroma scaling unit 615 (the operations of the luma mapping unit 610 and the chroma scaling unit 615 are collectively referred to as the luma mapping / chroma scaling (LMCS) process). The scaled chroma signal can be provided to the transform unit 630, and in particular, the transform unit 630 can perform an adaptive color transform on the chroma signal. The coefficients obtained from the above transformation are applied to the quantization unit 640 and quantized. Thus, lossy compression can be achieved, and the result of the lossy compression can be output as a bitstream 602 using a lossless compression method based on a multi-hypothesis CABAC unit 650.
[0153] On the other hand, in order to generate the coding loop, the result of the aforementioned lossy compression can undergo inverse quantization 645, inverse transform 635, and luma signal amplification 617 before entering the actual decoding process. The result of the aforementioned luma signal amplification can be provided to the internal combiner 607 together with the result of the selection 608 of at least one previously generated intra-frame predictive coding signal or inter-frame predictive coding signal. After undergoing inverse luma mapping 617, the result of the aforementioned internal combiner can be processed by a deblocking filter 660, a sample adaptive offset (SAO) 670, an adaptive loop filter (ALF), etc., for reproducing the image quality improvement process in the decoder. As described above, the result of the reproduction action in the decoder can be applied to the reference image buffer 690 and can be reused for predictive coding by the aforementioned inter-frame prediction unit 620.
[0154] The present invention can also be implemented or used in conjunction with the next-generation video codec, namely the enhanced compression model (ECM), developed by the Joint Video Experts Team (JVET) to improve H.266 / VVC. According to the aforementioned JVET standardization progress document, document specification 16 / 22 pages 20 CN 121488465 A number ISO / IEC JTC 1 / SC 29 / WG 5 N 190 (also document number JVET AC2025-v1), the enhanced compression model may include improved intra-frame predictive coding methods, improved inter-frame predictive coding methods, improved transform and transform coefficient coding methods, improved adaptive loop filtering methods, bilateral filtering methods, novel sample adaptive offset (SAO) methods for image quality improvement, extended entropy coding methods, and improved gradual decoding refresh (GDR) technology, and these technologies can be included in the present invention as background technology for the implementation of the present invention.
[0155] Features of the Invention
[0156] The present invention relates to a video encoding and decoding method and apparatus based on geometric segmentation prediction, which can be used in the field of video encoding and decoding including the above embodiments.
[0157] FIG7 is a conceptual diagram of a geometric segmentation prediction mode according to an embodiment of the present invention.The geometric segmentation prediction mode refers to the method of predicting and encoding block sample values described above, in which the blocks are re-divided into arbitrary geometric shapes instead of rectangular blocks, and preferably the shapes formed by dividing the rectangles with arbitrary diagonal lines are used for prediction and encoding.
[0158] In one embodiment of the present invention, the video bitstream may include a flag indicating whether the above-mentioned geometric segmentation prediction mode is used. Taking the second edition of the H.266 / VVC standard specification as an example, the above flag may be included in the "sps_gpm_enabled_flag" in the sequence parameter RBSP syntax in section 7.3.2.4. Of course, it is obvious that the above signal can provide at least 1 bit (or less than 1 bit when entropy coding is used) of information capacity at any other position in the bitstream.
[0159] According to one embodiment of the present invention, the above-mentioned geometric segmentation prediction mode may refer to dividing a block 710 into two or more regions 720, and applying the same or different prediction methods to each of the above-mentioned segmented regions 720. The above-mentioned prediction methods are not limited to this. For example, the prediction method described above can be an intra-frame prediction mode that references information from other blocks within the same frame, or an inter-frame prediction mode that references information from other intra-frames. Specific prediction methods included in the intra-frame or inter-frame modes may include a merge mode or an intra-block copy (IBC) mode. Furthermore, since one of the prediction methods described above may include a case where no prediction is performed, it may include an intra-frame coding method that encodes actual image information.
[0160] In one embodiment of the present invention, the intra-frame prediction mode described above may refer to the intra-frame prediction mode used for intra-frame prediction as defined in video codec standards such as H.266 / VVC. For example, the mode can be understood to include: specifying reference direction information from the block 710; deriving a prediction estimate when referencing at least one row of adjacent pixels formed on the interface of adjacent blocks in the reference direction; and deriving the prediction value of the block from the prediction estimate based on the predetermined reference direction using a method including linear or other algebraic methods and / or a predetermined kernel filtering method. This means that under the above geometric prediction mode, even if the two segmented regions 720 have the same intra-frame prediction mode, different intra-frame prediction information (e.g., reference directionality information) can be applied to each segmented region 720.
[0161] In one embodiment of the present invention, the above inter-frame prediction mode may refer to the inter-frame prediction mode used by inter-frame prediction as defined in video codec standards such as H.266 / VVC.For example, it can be understood that the above mode includes: in the encoding order (i.e. the decoding order), searching for image data similar to the above block 710 from the previous frame; determining the identifier of the reference frame for which the above similar image data is found, and specifying the reference vector (in the inter-frame prediction mode, it can be called the motion vector) of the above found point, and deriving the predicted value of the block based on the above reference vector. This means that in the above geometric prediction mode, even if two segmented regions 720 have the same inter-frame prediction mode, different inter-frame prediction information (e.g., the same or different reference frames apply different reference vectors) can be applied to each segmented region 720. Specification 17 / 22 pages 21 CN 121488465 A
[0162] According to an embodiment of the present invention, the above merging mode can refer to the merging mode prediction in inter-frame prediction as defined in video codec standards such as H.266 / VVC. The above-mentioned merging mode can refer to a method in an inter-frame prediction mode: this method does not allocate reference vector information for inter-frame prediction to a single block, but instead constructs a merging mode list consisting of a specified number of reference vector information in advance in a block set unit containing slices and coding tree units (CTUs), and calls and uses the corresponding information by specifying the index of the information included in the above list for individual predictions. This means that in the above-mentioned geometric prediction mode, even if two segmented regions 720 have the same merging mode, different merging mode indices can be applied to each partition 720.
[0163] According to an embodiment of the present invention, the above-mentioned IBC mode can refer to IBC mode prediction in intra-frame prediction as defined in video codec standards such as H.266 / VVC. The above-mentioned IBC mode can be understood as predicting the mode of a block of data by using previously encoded (and therefore, from the decoder's perspective, using previously decoded) block data as a reference, and its execution is similar to inter-frame prediction within the same frame. In other words, for example, the above mode can be understood as including: within the same frame, searching for image data similar to the aforementioned block 710 from previous blocks in the encoding order (i.e., decoding order), determining a reference vector (which can be called a block vector in IBC mode) pointing to the location where the similar image data was found, and deriving the predicted value of the block based on the aforementioned reference vector. This means that in the above geometric prediction mode, even if two segmented regions 720 also have the IBC mode, different reference vectors (i.e., block vectors) can be applied to each segmented region 720.
[0164] Furthermore, in the above-described IBC mode, the reference vector in the above-described IBC mode can be a list of reference vectors (which can be motion vectors or block vectors) used in blocks that are spatially adjacent to or close to the above-described block, and an index can be specified to specify one of the listed reference vectors. Furthermore, according to an embodiment, the residual vector between the reference vector specified by the index and the actual reference vector used in the application can be used by the encoder and / or decoder.
[0165] Taking the second edition of the H.266 / VVC standard specification as an example, the partitioning mode of the above-described region 720 can refer to the form specified by the value "merge_gpm_partition_idx" included in the merge data syntax in section 7.3.11.7. Furthermore, for the merge mode specified for each partitioned region, the index can refer to the content specified by the values "merge_gpm_idx0" and "merge_gpm_idx1" in the same section above. However, the above examples are examples of embodiments of the present invention based on the H.266 / VVC standard specification. The present invention can be applied to various video coding standards and is not limited to the above examples.
[0166] According to an embodiment of the present invention, the above-mentioned geometric segmentation prediction mode may refer to performing independent prediction for each of the above-mentioned segmentation regions 720 and performing weighted average prediction at the boundary 730 between regions. According to an embodiment of the present invention, the above-mentioned list of merging modes for geometric segmentation prediction may be composed of reference vector information for unidirectional prediction. The storage problem of segmentation region reference vector information will be explained below. FIG8 is a conceptual diagram showing the storage problem of segmentation region reference vector information. The conventional approach is to store the motion information 840 of the segmentation blocks within each segmentation region 820 generated by the segmentation block 810 for encoding and decoding. When each segmentation region 820 is predicted in a merging mode or an inter-frame mode with reference vector information, the corresponding motion information 840 is stored separately. Referring to FIG8(a), the figure shows the structure for storing the first motion information 840-1 about the first segmentation region 820-1 and the second motion information 840-2 about the second segmentation region 820-2. However, for the boundaries 830 between different regions, especially the segmented blocks 835 located on the aforementioned boundaries 830, a more efficient method is needed to store the reference vector information. In particular, as shown in FIG8(b), when at least one segmented region 820-3 uses intra-frame mode and lacks reference vector information 840-3, a reference vector information storage method is needed for this situation.
[0167] FIG9 is a conceptual diagram of a motion storage method for boundary region blocks according to the H.266 standard. The content shown in FIG9 is based on the second version of the H.266 standard released in April 2022.When storing reference vector information of a boundary region block, if reference vector information is stored in both the first and second segmented regions, and the reference vector information of each region in the boundary region block exists in different prediction lists 910, then all reference vector information 920 is stored; if the reference vector information of each region exists in the same prediction list, then only one reference vector information is stored. This invention provides a more optimized reference vector information storage method for this configuration.
[0168] According to one embodiment of the invention, if a segmented region (assuming the first segmented region mentioned above) is determined to be in a prediction mode that does not include reference vectors, then a zero vector or any vector can be stored as a reference vector for another segmented region (assuming the second segmented region mentioned above).
[0169] According to another embodiment of the invention, the reference vector of another region having reference vector information can be stored using the reference vector of the second segmented region, or a vector calculated using any formula using the reference vector of the first segmented region, or the reference vector of the second segmented region can be set to not exist. For example, if a regular inter-frame mode is selected in the first segmentation region and a reference vector is derived, the reference vector of the second segmentation region can be stored considering the reference vector (which may refer to a motion vector) used in the first segmentation region and related information. If template matching or IBC mode is selected in the first segmentation region and the reference vector is used in intra-frame mode, the reference vector of the second segmentation region can be stored considering the reference vector (which may refer to a block vector) used in intra-frame mode and related information.
[0170] According to another embodiment of the present invention, if the reference vector of the first segmentation region points to a block, or a block adjacent to or close to the block, the motion information of the second segmentation region can be determined using this.
[0171] For example, in inter-frame mode, a secondary reference vector (which may be a motion vector or a block vector) can be read from other frames specified by the reference vector (motion vector) in the inter-frame mode of the first segmentation region, and the read secondary reference vector or the vector obtained using any calculation formula can be stored.
[0172] For example, in IBC mode, reference vectors (which can be motion vectors or block vectors) used in blocks that are spatially adjacent to or close to the aforementioned block can be listed, and an index pointing to any of the listed reference vectors can be stored to indicate that the second segmentation region uses the aforementioned reference vector. As described above, when using reference vectors from other frames and / or blocks, the aforementioned reference vectors can be processed using any of the aforementioned calculation formulas, and residual vector information related to the aforementioned used reference vectors can be further stored.
[0173] According to one embodiment of the present invention, as a reference image index for a boundary region block, a zero index can be stored, or an index of other regions with reference vector information can be stored, or the reference image index can be set to non-existent. If template matching or IBC mode is selected and a reference vector is used in intra-frame mode, the reference image index of the boundary region block can be stored considering the reference vector (pointable to a block vector) used in the intra-frame mode and related information.
[0174] According to one embodiment of the present invention, if it is determined that at least one segmentation region in the first segmentation region and the second segmentation region is in a prediction mode that does not include a reference vector, the reference vector of the boundary region block can be stored as a zero vector or an arbitrary vector.
[0175] According to another embodiment of the present invention, as a reference vector for the boundary region block, a segmentation region with reference vector information, such as the reference vector of the first segmentation region or the second segmentation region, can be stored, or a modified reference vector can be derived and stored based on all, part or at least one of the reference vectors using any calculation formula.
[0176] According to another embodiment of the present invention, the reference vector of the boundary region block can be stored as a value indicating that it does not exist.
[0177] According to another embodiment of the present invention, if a reference vector is used in at least one of the first segmentation region or the second segmentation region, the reference vector of the second segmentation region can be stored taking into account the reference vector (which may be a motion vector or a block vector) and related information. If template matching or IBC mode is selected, the reference vector may be a block vector used in intra-frame mode; if inter-frame mode is selected, the reference vector may be a motion vector.
[0178] For example, if the first segmentation region selects IBC mode, while the second segmentation region only uses intra-frame coding, the reference vector (which may be a block vector) selected in the IBC mode of the first segmentation region can be stored as the reference vector of the boundary region block. Alternatively, the index of the list of reference vectors referenced from the surrounding blocks in the IBC mode of the first segmentation region can also be stored.
[0179] As another example, if both the first segmentation region and the second segmentation region select the IBC mode, then the reference vectors for the boundary region blocks can be stored from either of the two segmentation regions, or they can be stored as reference vectors processed by any calculation formula based on at least one of the aforementioned reference vectors. If the IBC mode of at least one of the two segmentation regions has a list of reference vectors for the surrounding blocks, then the index of the reference vector list used by at least one of the at least one segmentation region can be stored. The index points to the reference vector list, which may be the same as or different from the segmentation region.Furthermore, reference vector lists of at least two or more segmented regions can be merged, and the index of the merged list can be stored.
[0180] According to another embodiment of the present invention, if the above-mentioned reference vector information corresponds to the motion vector of the inter-frame mode and further includes information of the reference image, it can further include reference image index information. In this case, the above-mentioned reference image index information can be derived using an implementation method similar to that of the above-mentioned reference vector. For example, a zero index can be stored for the index of the reference image of the above-mentioned boundary region block, or a reference image index from at least one segmented region can be stored in the reference image index of the boundary region block, or the reference image index of the boundary region can be set to not exist. Alternatively, if a reference image index is stored in the block indicated by the reference vector of at least one segmented region or in its spatially surrounding blocks, the index can be utilized. Various embodiments not limited to those described above can be used to implement the method for deriving the above-mentioned reference image index information, and it can be consistent with or inconsistent with the method for deriving the above-mentioned reference vector.
[0181] According to an embodiment of the present invention, if at least one segmented region does not include reference vector information, the predicted value of the boundary region block can be averaged using the value predicted by the mode used by at least one region in the first segmented region or the second segmented region; or one of the modes used by the segmented region can be selected and the value predicted by that mode can be used; or the values predicted by each mode can be weighted and averaged before use. When using an intra-frame or inter-frame mode based on reference vectors, the previously generated reconstructed block values in the same or different frames indicated by the reference vectors can be referenced and used.
[0182] According to an embodiment of the present invention, when using the average of weighted values, the weighted values can be determined according to the encoded information content including the prediction mode and / or reference vector information of the surrounding blocks. For example, the weighted values of each predicted value can be determined according to whether the surrounding block is an intra-frame mode; or, for example, the weighted values of each predicted value can be determined according to the distance between the surrounding block and the adjacent boundary region block and the type of prediction mode. When using an intra-frame or inter-frame mode based on reference vectors, the weighted values can be determined according to the type of information encoded in the same or different frames pointed to by the reference vectors.
[0183] According to another embodiment of the present invention, if at least one segmented region included in the above block does not include reference vector information, a flag or index indicating whether an intra-frame coding mode is used, or a flag or index indicating whether a merge / inter-frame coding mode is used, can be transmitted. When an intra-frame coding mode is used, a flag or index of the intra-frame coding mode type can be transmitted, and when using an intra-frame coding mode, a pre-specified intra-frame coding mode can be used.In another embodiment of the present invention, if bidirectional reference vector information is used in at least one segmented region included in the above-mentioned block, the reference vector of the boundary region block can specify any segmented region and store the reference vector of the segmented region; or, store the reference vector determined based on the prediction list or reference image index of each region; or, store the reference vector by analyzing the position, reference image index, reference vector, etc. of the boundary region block; or, store the vector by weighting the average of more than one reference vector in terms of direction; or store the motion vectors with the same prediction list or reference image index after weighting the average; or, adjust the reference vector through the same reference image and store it after weighting the average.
[0184] According to one embodiment of the present invention, the step of “determining any region as the reference vector of the above-mentioned boundary region block and storing the reference vector of the above-mentioned region” can constitute a feature of the present invention. In addition, any region can be determined by the reference image index of the boundary region block and the index of the region can be stored; or, a zero index can be stored for the reference image index of the boundary region block; or, the lowest reference image index can be stored. The step of “determining any region as the reference image index of the boundary region block and storing the index of the region” can constitute a feature of the present invention.
[0185] According to an embodiment of the present invention, if bidirectional reference vector information is used in at least one segmentation region included in the block, the bidirectional reference vector is stored preferentially as the reference vector of the boundary region block, rather than the unidirectional reference vector; the index of the bidirectional reference vector information is stored preferentially as the reference image index of the boundary region block, which is a feature of the present invention. FIG10 is a conceptual diagram of the preferential storage of bidirectional motion vectors and indexes according to an embodiment of the present invention. Referring to FIG10, for the boundary region block 1035 located on the boundary line 1030 between the first segmentation region 1020-1 and the second segmentation region 1020-2 in the geometric segmentation block 1010, the figure shows a schematic diagram of a structure for storing vectors and / or indexes, which allows the boundary region block to use the same vector 1040-2 as the second segmentation region 1020-2.
[0186] FIG11 is a conceptual diagram of adding unidirectional reference vector information to the merging mode list according to an embodiment of the present invention.In another embodiment of the present invention, when configuring a merging mode list for each segmented region, as shown in FIG11(a), a merging mode list 1110 including unidirectional reference vector information 1120 and a merging mode list 1111 including bidirectional reference vector information 1130 can be configured respectively; or, as shown in FIG11(b), the merging mode including bidirectional reference vector information 1130 is given high priority and configured as a merging mode list 1112; or, the bidirectional reference vector information 1130 is first added to the merging mode list 1112, and then the unidirectional reference vector information 1120 is added to the blank space; or, as shown in FIG11(c), the merging mode including unidirectional reference vector information 1120 is given high priority and configured as a list 1113; or, the unidirectional reference vector information 1120 is first added to the merging mode list 1113, and then the bidirectional reference vector information 1130 is added to the blank space.
[0187] In another embodiment of the present invention, the application of geometric segmentation prediction technology can be determined by considering the horizontal or vertical size of the block, its area, the number of pixels within the block, or information about adjacent blocks. In this case, it can be determined whether geometric segmentation prediction uses a merge mode, an inter-frame mode, or an intra-frame mode. Furthermore, it can be determined whether geometric segmentation prediction uses unidirectional reference vector information or bidirectional reference vector information. Furthermore, it can be determined whether geometric segmentation prediction technology should be combined with other prediction technologies.
[0188] According to one embodiment of the present invention, a first threshold for the horizontal size of the block and a second threshold for the vertical size of the block can be specified. According to a specific embodiment, if the horizontal size of the block is greater than the first threshold, and / or the vertical size of the block is greater than the second threshold, the use of intra-frame mode can be restricted. In another embodiment, if the horizontal size of the block is less than the first threshold, and / or if the vertical size of the block is less than the second threshold, the use of bidirectional reference vector information can be restricted. In yet another embodiment, a third threshold for the number of pixels included in the block (i.e., the pixel area of the block) can be specified, and if the number of pixels within the block is greater than the third threshold, the combination of geometric segmentation prediction with other prediction technologies can be restricted. The deterministic nature of the invention is described on pages 21 / 22 of the specification, CN 121488465 A, and constitutes a feature of the present invention.
[0189] As described above, the feature is that it stores boundary region information including the reference vector of the aforementioned boundary region block and / or the reference image index of the aforementioned boundary region, so as to be referenced in the encoding and / or decoding process of subsequent other blocks. For example, when the aforementioned boundary region block is referenced intra-frame or inter-frame in the subsequent encoding / decoding process, the step of determining the encoding method of the boundary region block can constitute a feature of the present invention.
[0190] The present invention has been described above with reference to the accompanying drawings and embodiments, but as described above, this does not mean that the scope of protection of the present invention is limited to the drawings or embodiments disclosed above.Those skilled in the art should understand that various modifications and alterations can be made to this invention without departing from the inventive concept and scope set forth in the claims of this patent. Instruction manual, page 22 / 22; Figure 1 (CN 121488465 A); Figure 2 (CN 121488465 A); Figure 3 (CN 121488465 A); Figure 4 (CN 121488465 A); Figure 5 (CN 121488465 A); Figure 6 (CN 121488465 A); Figure 7 (CN 121488465 A); Figure 8 (CN 121488465 A); Figure 9 (CN 121488465 A); Figure 10 (CN 121488465 A); Figure 11 (CN 121488465 A). The instruction manual includes attached figures on page 10 / 10, page 36, CN 121488465 A.
Claims
1. A decoding method, as a video decoding method based on geometric segmentation prediction, characterized in that, include: The steps to read geometric segmentation pattern information about the geometric segmentation of blocks from the encoded bitstream; The steps for determining the first segmentation region and the second segmentation region based on the geometric segmentation pattern information; Read the first prediction information from the encoded bitstream; Read the second prediction information from the encoded bitstream; as well as The steps are as follows: generating a first prediction block for the first segmented region based on the first prediction information, generating a second prediction block for the second segmented region based on the second prediction information, and combining the first prediction block and the second prediction block to generate a final prediction block. The first prediction information and the second prediction information are different from each other.
2. The decoding method according to claim 1, wherein, The steps for generating the final prediction block include: The step of determining the boundary region including the boundary line between the first segmented region and the second segmented region based on the geometric segmentation pattern information; The step of generating the boundary region prediction block; The step of generating a final prediction block by combining the first prediction block, the second prediction block, and the boundary region prediction block; and The step of storing boundary region information about the predicted boundary region block.
3. The decoding method according to claim 2, characterized in that, The boundary region prediction block is generated based on the weighted average of the first prediction block and the second prediction block.
4. The decoding method according to claim 3, characterized in that, The weighting value is determined based on at least one of the following: the first prediction information, the second prediction information, the prediction information of the surrounding blocks, whether the prediction information of the surrounding blocks is intra-frame prediction information, the distance between the surrounding blocks and the generated boundary region, and the reference vector information of the surrounding blocks.
5. The decoding method according to claim 2, characterized in that, The first prediction information includes first reference vector information. The second prediction information does not include reference vector information. The boundary region information is generated based on the first reference vector.
6. The decoding method according to claim 2, characterized in that, The first prediction information regarding the first segmented region includes first reference vector information. The second prediction information regarding the second segmented region includes second reference vector information. The boundary region information is generated based on at least one of the first prediction information and the second prediction information.
7. The decoding method according to claim 6, characterized in that, The first prediction information pertains to the intra-block copying mode. The first reference vector represents a block vector. The boundary region information is generated based on the first prediction information.
8. The decoding method according to claim 7, characterized in that, The first prediction information includes the first index of the candidate list of intra-frame block copy reference vectors.
9. The decoding method according to claim 7, characterized in that, The second prediction information pertains to the intra-frame block copy mode, and the second reference vector represents the block vector. The boundary region information is generated based on the first prediction information and the second prediction information.
10. The decoding method according to claim 9, characterized in that, The first prediction information includes the first index of the candidate list of intra-frame block copy reference vectors. The second prediction information includes the second index of the candidate list of intra-frame block copy reference vectors. The boundary region information is generated based on third index information, wherein the third index information is determined based on at least one of the first index and the second index.
11. The decoding method according to claim 4, characterized in that, The first prediction information is about the inter-frame mode. The first reference vector represents the motion vector. The boundary region information is generated based on the first prediction information.
12. The decoding method according to claim 11, characterized in that, The first prediction information includes a first index based on a first reference vector candidate list under the merging pattern.
13. The decoding method according to claim 11, characterized in that, The second prediction information is about inter-frame patterns. The second reference vector represents the motion vector. The boundary region information is generated based on the first prediction information and the second prediction information.
14. The decoding method according to claim 13, characterized in that, The first prediction information includes a first index of a first reference vector candidate list based on a merging pattern. The second prediction information includes a second index based on the second reference vector candidate list under the merging pattern. The boundary region information is generated based on third index information, wherein the third index information is determined based on at least one of the first index and the second index.
15. The decoding method according to claim 13, characterized in that, The first prediction information includes a bidirectional reference vector. The second prediction information includes a one-way reference vector. The boundary region information is generated based on the first prediction information.
16. The decoding method according to any one of claims 6, 7, or 11, characterized in that, The boundary region information is generated based on a reference vector obtained by weighted averaging of the first reference vector and the second reference vector.
17. A coding method, as a video coding method based on geometric segmentation prediction, characterized in that, include: The steps for determining the first and second segmented regions based on the geometric segmentation information of the blocks; The steps to determine the first predicted information about the first segmented region; The steps for determining second prediction information about the second segmented region; as well as The step of encoding the first prediction information, the second prediction information, and the combined information into a bitstream. The first prediction information and the second prediction information are different from each other.
18. The encoding method according to claim 17, characterized in that, Also includes: The step of setting a boundary region including the boundary lines of the first segmented region and the second segmented region; And, the step of storing boundary region information about the predicted boundary region block.
19. The encoding method according to claim 18, characterized in that, The first prediction information includes the first index of the candidate list of intra-frame block copy reference vectors. The second prediction information includes the second index of the candidate list of intra-frame block copy reference vectors. The boundary region information includes third index information determined based on at least one of the first index and the second index.
20. A decoder device, as a video decoder device based on geometric segmentation prediction, characterized in that, include: The bitstream parsing unit is used to read geometric segmentation pattern information, first prediction information and second prediction information about the geometric segmentation of blocks from the encoded bitstream; The geometric segmentation prediction unit determines a first segmentation region and a second segmentation region based on the geometric segmentation pattern information, generates a first prediction block about the first segmentation region based on the first prediction information, generates a second prediction block about the second segmentation region based on the second prediction information, and generates a final prediction block by combining the first prediction block and the second prediction block. as well as The memory stores the decoded image information, including the final prediction block. The first prediction information and the second prediction information are different from each other.
