An image prediction method and apparatus

Abstract

This application discloses an image prediction method and related products. The method includes: parsing a bitstream to obtain a multi-hypothesis information index for a current image block to be decoded; obtaining first multi-hypothesis information corresponding to the current image block from a multi-hypothesis information list based on the multi-hypothesis information index, wherein the first multi-hypothesis information includes a motion information index and a first identifier, the motion information index indicating the multi-hypothesis motion information of the current image block to be decoded, the first identifier indicating a first partitioning method of the multi-hypothesis method, and the first multi-hypothesis information also including parameters of the first partitioning method; and processing the predicted image blocks of multiple hypotheses using the first partitioning method based on the first identifier and the parameters of the first partitioning method to obtain the predicted image block of the current image block to be decoded. In this way, different image blocks can use different partitioning methods to obtain the predicted image blocks, thereby improving the efficiency of image prediction and reducing decoding time.

Description

Technical Field

[0001] This application relates to the field of video encoding and decoding technology, and more specifically, to an image prediction method and apparatus. Background Technology

[0002] Digital video capabilities can be incorporated into a wide variety of devices, including digital television, digital live broadcasting systems, wireless broadcasting systems, personal digital assistants (PDAs), laptops or desktop computers, tablets, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radio phones (so-called "smartphones"), video conferencing devices, video streaming devices, and the like. Digital video devices implement video compression techniques, such as those described in standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 Part 10 High-Efficiency Video Coding (AVC), the H.265 / High Efficiency Video Coding (HEVC) standard, and extensions to such standards. By implementing such video compression techniques, video devices can transmit, receive, encode, decode, and / or store digital video information more efficiently.

[0003] Video compression techniques perform spatial (intra-image) prediction and / or temporal (inter-image) prediction to reduce or remove inherent redundancy in video sequences. For block-based video coding, a video strip (i.e., a video frame or a portion of a video frame) can be divided into several image blocks, which may also be referred to as tree blocks, coding units (CUs), and / or coding nodes. Image blocks in an intra-frame coded (I) strip of an image are encoded using spatial predictions relative to reference samples in adjacent blocks within the same image. Image blocks in an inter-frame coded (P or B) strip of an image can use spatial predictions relative to reference samples in adjacent blocks within the same image or temporal predictions relative to reference samples in other reference images. An image may be referred to as a frame, and a reference image may be referred to as a reference frame.

[0004] Various video coding standards, including the HEVC standard, have proposed predictive coding modes for image blocks, which predict the current block to be encoded based on already encoded video data blocks. In intra-frame prediction mode, the current block is predicted based on one or more previously decoded neighboring blocks in the same image as the current block; in inter-frame prediction mode, the current block is predicted based on already decoded blocks in different images.

[0005] Current inter-frame prediction methods are inefficient, which affects the execution time of the entire decoding process. Summary of the Invention

[0006] This application provides an image prediction method and apparatus that can improve the efficiency of image prediction and thus reduce the time of the entire decoding process.

[0007] A first aspect provides an image prediction method, the method comprising: parsing a bitstream to obtain a multi-hypothesis information index of a current image block to be decoded; obtaining first multi-hypothesis information corresponding to the current image block from a multi-hypothesis information list based on the multi-hypothesis information index, the first multi-hypothesis information including a motion information index and a first identifier, the motion information index indicating multi-hypothesis motion information of the current image block to be decoded, the first identifier indicating a first partitioning method of the multi-hypothesis method, and the first multi-hypothesis information further including parameters of the first partitioning method; performing motion compensation based on the motion information encoded by the multi-hypothesis of the current image block to be decoded to obtain a predicted image block of multiple hypotheses; and processing the predicted image blocks of the multiple hypotheses using the first partitioning method based on the first identifier and the parameters of the first partitioning method to obtain a predicted image block of the current image block to be decoded.

[0008] In one possible implementation, the list of multiple hypothesis information includes at least one first multiple hypothesis information and at least one second multiple hypothesis information. The second multiple hypothesis information includes a motion information index and a second identifier. The motion information index of the second multiple hypothesis information indicates the multiple hypothesis motion information of an image patch, and the second identifier indicates a second partitioning method of the multiple hypothesis method. The second multiple hypothesis information also includes parameters of the second partitioning method.

[0009] In one possible implementation, the first identifier and the second identifier are different values ​​of the same flag bit.

[0010] In one possible implementation, the first partitioning method is a triangular partitioning, the parameters of which indicate the partitioning direction of the triangular partitioning, and the second partitioning method is a square partitioning, the parameters of which indicate the weighting coefficients of the square partitioning.

[0011] It should be understood that the first and second partitioning methods are different. Each partitioning method is used to divide the predicted image patches for multiple hypotheses, that is, to divide the predicted image patches for the original hypothesis and the predicted image patches for the additional hypotheses. For specific methods and approaches to partitioning the predicted image patches using the multiple hypotheses method, please refer to the content of JVET-K0144 proposal (triangular partitioning) and JVET-K0257 proposal (square partitioning).

[0012] In one possible implementation, the first partitioning method is a triangular partitioning, and the parameters of the first partitioning method indicate the partitioning direction of the triangular partitioning; or the first partitioning method is a square partitioning, and the parameters of the first partitioning method indicate the weighting coefficients of the square partitioning.

[0013] In one possible implementation, the method further includes: obtaining the multi-hypothesis motion information of the current image block to be decoded from the candidate motion information list of the current image block to be decoded based on the motion information index of the first multi-hypothesis information, wherein the multi-hypothesis motion information of the current image block to be decoded includes the motion information of the original hypothesis and the motion information of the additional hypothesis.

[0014] In one possible implementation, the motion information index of the first multiple hypothesis information includes a first index and a second index. The process of obtaining the multiple hypothesis motion information of the current image block to be decoded from the candidate motion information list of the current image block to be decoded based on the motion information index of the first multiple hypothesis information includes: obtaining the motion information of the original hypothesis from the first candidate motion information list of the current image block based on the first index, and obtaining the motion information of the additional hypothesis from the second candidate motion information list of the current image block based on the second index.

[0015] In this way, by using the above method, since the bitstream carries a partitioning identifier indicating the multi-hypothesis method, the decoding end can determine which partitioning method to use in the multi-hypothesis method to predict the image. That is, different partitioning methods can be used to obtain the prediction blocks of different image blocks. In this way, the image blocks can use a partitioning method that is more suitable for their own characteristics, thereby improving the efficiency of image prediction and reducing the decoding time.

[0016] In a second aspect, an image prediction apparatus is provided, the apparatus comprising a module for performing the method in any of the implementations of the first aspect described above.

[0017] Thirdly, an image prediction apparatus is provided, comprising: a non-volatile memory coupled to each other and a processor, the processor calling program code stored in the memory to perform some or all of the steps of the method in any implementation of the first aspect.

[0018] Fourthly, a computer-readable storage medium is provided that stores program code, wherein the program code includes instructions for performing some or all of the steps of the method in any implementation of the first aspect.

[0019] Fifthly, a computer program product is provided that, when the computer program product is run on a computer, causes the computer to execute instructions for some or all of the steps of the method in any implementation of the first aspect.

[0020] A sixth aspect provides an image prediction device, comprising: a memory for storing video data in the form of a bitstream, the video data including one or more image blocks; a video decoder for parsing the bitstream to obtain a multi-hypothesis information index of the current image block to be decoded; obtaining first multi-hypothesis information corresponding to the current image block from a multi-hypothesis information list based on the multi-hypothesis information index, the first multi-hypothesis information including a motion information index and a first identifier, the motion information index indicating multi-hypothesis motion information of the current image block to be decoded, the first identifier indicating a first partitioning method of the multi-hypothesis method, and the first multi-hypothesis information further including parameters of the first partitioning method; performing motion compensation based on the motion information encoded by the multi-hypothesis of the current image block to be decoded to obtain a predicted image block of multiple hypotheses; and processing the predicted image blocks of the multiple hypotheses using the first partitioning method based on the first identifier and the parameters of the first partitioning method to obtain a predicted image block of the current image block to be decoded.

[0021] In one possible implementation, the list of multiple hypothesis information includes at least one first multiple hypothesis information and at least one second multiple hypothesis information. The second multiple hypothesis information includes a motion information index and a second identifier. The motion information index of the second multiple hypothesis information indicates the multiple hypothesis motion information of an image patch, and the second identifier indicates a second partitioning method of the multiple hypothesis method. The second multiple hypothesis information also includes parameters of the second partitioning method.

[0022] In one possible implementation, the first identifier and the second identifier are different values ​​of the same flag bit.

[0023] In one possible implementation, the first partitioning method is a triangular partitioning, the parameters of which indicate the partitioning direction of the triangular partitioning, and the second partitioning method is a square partitioning, the parameters of which indicate the weighting coefficients of the square partitioning.

[0024] It should be understood that the first and second partitioning methods are different. Each partitioning method is used to divide the predicted image patches for multiple hypotheses, that is, to divide the predicted image patches for the original hypothesis and the predicted image patches for the additional hypotheses. For specific methods and approaches to partitioning the predicted image patches using the multiple hypotheses method, please refer to the content of JVET-K0144 proposal (triangular partitioning) and JVET-K0257 proposal (square partitioning).

[0025] In one possible implementation, the first partitioning method is a triangular partitioning, and the parameters of the first partitioning method indicate the partitioning direction of the triangular partitioning; or the first partitioning method is a square partitioning, and the parameters of the first partitioning method indicate the weighting coefficients of the square partitioning.

[0026] In one possible implementation, the video decoder is further configured to: obtain the multi-hypothesis motion information of the current image block to be decoded from the candidate motion information list of the current image block to be decoded based on the motion information index of the first multi-hypothesis information, wherein the multi-hypothesis motion information of the current image block to be decoded includes the motion information of the original hypothesis and the motion information of the additional hypothesis.

[0027] In one possible implementation, the motion information index of the first multiple hypothesis information includes a first index and a second index. In the aspect of obtaining the multiple hypothesis motion information of the current image block to be decoded from the candidate motion information list of the current image block according to the motion information index of the first multiple hypothesis information, the video decoder is further configured to: obtain the motion information of the original hypothesis from the first candidate motion information list of the current image block according to the first index, and obtain the motion information of the additional hypothesis from the second candidate motion information list of the current image block according to the second index.

[0028] In this way, by using the above-mentioned equipment, since the bitstream carries a partitioning identifier indicating the multi-hypothesis method, the decoding end can determine which partitioning method to use in the multi-hypothesis method to predict the image. That is, different partitioning methods can be used to obtain the prediction blocks of different image blocks. In this way, the image blocks can use a partitioning method that is more suitable for their own characteristics, thereby improving the efficiency of image prediction and reducing the decoding time. Attached Figure Description

[0029] Figure 1 This is a conceptual block diagram of coding system 10;

[0030] Figure 2 This is a schematic block diagram of an example of a video encoder 20;

[0031] Figure 3 This is a schematic block diagram of an example of video decoder 30;

[0032] Figure 4 This is an illustrative diagram of an example of a video coding system 40;

[0033] Figure 5 This is a schematic diagram of an apparatus 500 for implementing the method of this application;

[0034] Figure 6 This is a schematic diagram of an image prediction method;

[0035] Figure 7 This is a schematic diagram of an image prediction device. Detailed Implementation

[0036] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0037] The following description, with reference to the accompanying drawings, illustrates specific embodiments of the present invention and application examples using these embodiments. Video encoding generally refers to processing a sequence of images that form a video or video sequence. In the field of video encoding, the terms "picture," "frame," or "image" can be used synonymously. The video encoding used in this application (or this disclosure) refers to video encoding or video decoding. Video encoding is performed on the source side and typically involves processing (e.g., by compression) the original video picture to reduce the amount of data required to represent the video picture (thus storing and / or transmitting it more efficiently). Video decoding is performed on the destination side and typically involves inverse processing relative to the encoder to reconstruct the video picture. The "encoding" of video pictures (or collectively referred to as pictures, as explained below) involved in the embodiments should be understood as involving the "encoding" or "decoding" of a video sequence. The combination of encoding and decoding portions is also referred to as encoding and decoding (encoding and decoding).

[0038] In lossless video coding, the original video images can be reconstructed, meaning the reconstructed video images have the same quality as the original (assuming no transmission loss or other data loss during storage or transmission). In lossy video coding, further compression is performed, for example, through quantization, to reduce the amount of data required to represent the video images. However, the decoder cannot fully reconstruct the video images, meaning the quality of the reconstructed video images is lower or worse than the original.

[0039] Several H.261 video coding standards fall under the category of "lossy hybrid video coding and decoding" (i.e., combining spatial and temporal prediction in the sample domain with 2D transform coding in the transform domain for applying quantization). Each frame of a video sequence is typically segmented into a non-overlapping set of blocks, usually encoded at the block level. In other words, the encoder typically processes, i.e., encodes the video at the block (video block) level, for example, by generating prediction blocks through spatial (intra-frame) and temporal (inter-frame) predictions, subtracting the prediction blocks from the current block (the currently processed or pending block) to obtain residual blocks, transforming and quantizing the residual blocks in the transform domain to reduce the amount of data to be transmitted (compressed), while the decoder applies the inverse processing relative to the encoder to the encoded or compressed blocks to reconstruct the current block for representation. Additionally, the encoder replicates the decoder processing loop, causing the encoder and decoder to generate the same predictions (e.g., intra-frame and inter-frame predictions) and / or reconstructions for processing, i.e., encoding subsequent blocks.

[0040] As used herein, the term "block" can be a portion of a picture or frame. For ease of description, embodiments of the invention are described with reference to High-Efficiency Video Coding (HEVC), developed by the Joint Collaboration Team on Video Coding (JCT-VC) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG). Those skilled in the art will understand that embodiments of the invention are not limited to HEVC. "CU," "PU," and "TU" can refer to each other. In HEVC, a CTU is split into multiple CUs using a quadtree structure represented as a coding tree. At the CU level, a decision is made as to whether to use inter-picture (temporal) or intra-picture (spatial) prediction to encode picture regions. Each CU can be further split into one, two, or four PUs depending on the PU splitting type. The same prediction process is applied within a PU, and relevant information is transmitted to the decoder on a PU basis. After obtaining residual blocks through a prediction process based on PU splitting type, the CU can be segmented into transform units (TUs) according to other quadtree structures similar to those used for the CU. In recent developments in video compression technology, quad-tree and binary tree (QTBT) frame segmentation is used to divide coded blocks. In the QTBT block structure, the CU can be square or rectangular. In XXX, the coding tree unit (CTU) is first segmented by a quadtree structure. The leaf nodes of the quadtree are further segmented by a binary tree structure. The leaf nodes of the binary tree are called coding units (CUs), and these segments are used for prediction and transform processing without any further segmentation. This means that the CU, PU, ​​and TU have the same block size in the QTBT coding block structure. Simultaneously, multiple segmentation, such as ternary tree segmentation, is proposed to be used with the QTBT block structure.

[0041] The following is based on Figures 1 to 4 Embodiments describing encoder 20, decoder 30, and encoding / decoding systems 10 and 40 (based on...) Figure 6 (Before describing the embodiments of the present invention in more detail).

[0042] Figure 1For the purpose of illustrating a conceptual or schematic block diagram of an exemplary encoding system 10, such as a video encoding system 10 that may utilize the technology of this application (this disclosure), the encoder 20 (e.g., video encoder 20) and decoder 30 (e.g., video decoder 30) of the video encoding system 10 represent device instances that can be used to perform techniques for image prediction according to various examples described in this application. Figure 1 As shown, the encoding system 10 includes a source device 12 for providing encoded data 13, such as an encoded image 13, to a destination device 14 for example, decoding encoded data 13.

[0043] The source device 12 includes an encoder 20, and optionally may include an image source 16, such as a preprocessing unit 18 of an image preprocessing unit 18, and a communication interface or communication unit 22.

[0044] Image source 16 may include or may be any type of image capture device, for example, for capturing real-world images, and / or any type of image or commentary (for screen content encoding, some text on the screen is also considered part of the image or picture to be encoded) generation device, such as a computer graphics processor for generating computer-animated images, or any type of device for acquiring and / or providing real-world images, computer-animated images (e.g., screen content, virtual reality (VR) images), and / or any combination thereof (e.g., augmented reality (AR) images).

[0045] A digital image is, or can be viewed as, a two-dimensional array or matrix of sampled points with luminance values. These sampled points in the array are also called pixels (short for picture element) or pels. The number of sampled points in the array or image along the horizontal and vertical directions (or axes) defines the image's size and / or resolution. To represent color, three color components are typically used; that is, an image can be represented as or contain three sampled arrays. In RBG format or color space, an image includes corresponding red, green, and blue sampled arrays. However, in video coding, each pixel is typically represented in a luma / chroma format or color space, such as YCbCr, including the luminance component indicated by Y (sometimes also indicated by L) and two chroma components indicated by Cb and Cr. The luminance (abbreviated as luma) component Y represents the luminance or grayscale level intensity (e.g., both are the same in grayscale images), while the two chroma (abbreviated as chroma) components Cb and Cr represent the chroma or color information components. Accordingly, a YCbCr format image includes a luminance sample array of luminance values ​​(Y) and two chrominance sample arrays of chrominance values ​​(Cb and Cr). An RGB format image can be converted or transformed to YCbCr format, and vice versa; this process is also known as color transformation or conversion. If the image is black and white, it may only include the luminance sample array.

[0046] Image source 16 (e.g., video source 16) can be, for example, a camera for capturing images, a memory for image storage, including or storing previously captured or generated images, and / or any type of (internal or external) interface for acquiring or receiving images. The camera can be, for example, a local or integrated camera within the source device, and the memory can be local or, for example, integrated memory within the source device. The interface can be, for example, an external interface for receiving images from an external video source, such as an external image capture device, like a camera, external memory, or an external image generation device, such as an external computer graphics processor, computer, or server. The interface can be any type of interface according to any proprietary or standardized interface protocol, such as a wired or wireless interface, or an optical interface. The interface for acquiring image data 17 can be the same interface as communication interface 22 or part of communication interface 22.

[0047] Unlike the preprocessing unit 18 and the processing performed by the preprocessing unit 18, the image or image data 17 (e.g., video data 16) can also be referred to as the raw image or raw image data 17.

[0048] The preprocessing unit 18 is used to receive (raw) image data 17 and perform preprocessing on the image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. For example, the preprocessing performed by the preprocessing unit 18 may include retouching, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It is understood that the preprocessing unit 18 may be an optional component.

[0049] Encoder 20 (e.g., video encoder 20) is used to receive preprocessed image data 19 and provide encoded image data 21 (details will be further described below, e.g., based on...) Figure 2 or Figure 4 In one instance, encoder 20 can be used during the encoding process to carry an identifier indicating the partitioning method of the multi-hypothesis method and the parameters of that partitioning method in the bitstream of the image block, as detailed in the corresponding paragraphs below.

[0050] The communication interface 22 of the source device 12 can be used to receive encoded image data 21 and transmit it to other devices, such as the destination device 14 or any other device, for storage or direct reconstruction, or for processing the encoded image data 21 before correspondingly storing the encoded data 13 and / or transmitting the encoded data 13 to other devices, such as the destination device 14 or any other device for decoding or storage.

[0051] Destination device 14 includes decoder 30 (e.g., video decoder 30), and may also optionally include communication interface or communication unit 28, post-processing unit 32 and display device 34.

[0052] The communication interface 28 of the destination device 14 is used, for example, to receive encoded image data 21 or encoded data 13 directly from the source device 12 or any other source, such as a storage device, which is an encoded image data storage device.

[0053] Communication interfaces 22 and 28 can be used to transmit or receive encoded image data 21 or encoded data 13 via a direct communication link between source device 12 and destination device 14 or via any type of network. The direct communication link is, for example, a direct wired or wireless connection, and any type of network is, for example, a wired or wireless network or any combination thereof, or any type of private or public network or any combination thereof.

[0054] The communication interface 22 can be used, for example, to encapsulate the encoded image data 21 into a suitable format, such as a packet, for transmission over a communication link or communication network.

[0055] The corresponding part of the communication interface 22, the communication interface 28, can be used, for example, to decapsulate the encoded data 13 to obtain the encoded image data 21.

[0056] Both communication interface 22 and communication interface 28 can be configured as unidirectional communication interfaces, such as Figure 1 The arrow pointing from the source device 12 to the destination device 14 for encoded image data 13 is indicated, or configured as a bidirectional communication interface, and can be used, for example, to send and receive messages to establish a connection, acknowledge and exchange any other information related to the communication link and / or, for example, the data transmission of encoded image data.

[0057] Decoder 30 is used to receive encoded image data 21 and provide decoded image data 31 or decoded image 31 (details will be further described below, for example, based on...) Figure 3 or Figure 5 In one instance, decoder 30 can be used to implement the image prediction method described in this application.

[0058] The post-processor 32 of the destination device 14 is used to post-process the decoded image data 31 (also known as reconstructed image data), for example, decoded image 131, to obtain post-processed image data 33, for example, post-processed image 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color correction, retouching or resampling, or any other processing, for example, preparing the decoded image data 31 for display by the display device 34.

[0059] The display device 34 of the destination device 14 is used to receive the post-processed image data 33 to display the image to, for example, a user or viewer. The display device 34 can be or may include any class of displays for presenting the reconstructed image, such as integrated or external displays or monitors. For example, the display may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other class of display.

[0060] Although Figure 1Source device 12 and destination device 14 are illustrated as separate devices, but device embodiments may also include the functionality of both source device 12 and destination device 14, or both; that is, the functionality of source device 12 or its corresponding functionality and the functionality of destination device 14 or its corresponding functionality. In such embodiments, the functionality of source device 12 or its corresponding functionality and the functionality of destination device 14 or its corresponding functionality may be implemented using the same hardware and / or software, or using separate hardware and / or software, or any combination thereof.

[0061] Based on the description, it is obvious to those skilled in the art that the functionality of different units is different. Figure 1 The presence and (accurate) division of the functionality of the source device 12 and / or destination device 14 shown may vary depending on the actual device and application.

[0062] Encoder 20 (e.g., video encoder 20) and decoder 30 (e.g., video decoder 30) can both be implemented as any of a variety of suitable circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof. If the technology is implemented in part in software, the device can store software instructions in a suitable non-transitory computer-readable storage medium, and one or more processors can be used to execute the instructions in hardware to perform the technology of this disclosure. Any of the foregoing (including hardware, software, combinations of hardware and software, etc.) can be considered as one or more processors. Each of video encoder 20 and video decoder 30 can be included in one or more encoders or decoders, any of which can be integrated as part of a combined encoder / decoder (encoder-decoder) in the corresponding device.

[0063] The source device 12 may be referred to as a video encoding device or video encoding apparatus. The destination device 14 may be referred to as a video decoding device or video decoding apparatus. The source device 12 and the destination device 14 may be instances of a video encoding device or video encoding apparatus.

[0064] The source device 12 and the destination device 14 may include any of a variety of devices, including any category of handheld or stationary devices, such as laptops or notebooks, mobile phones, smartphones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, video game consoles, video streaming devices (such as content service servers or content distribution servers), broadcast receiver devices, broadcast transmitter devices, etc., and may or may not use any category of operating system.

[0065] In some cases, source device 12 and destination device 14 can be equipped for wireless communication. Therefore, source device 12 and destination device 14 can be wireless communication devices.

[0066] In some cases, Figure 1 The video encoding system 10 shown is merely an example, and the technology of this application can be applied to video encoding setups (e.g., video encoding or video decoding) that do not necessarily involve any data communication between encoding and decoding devices. In other instances, data may be retrieved from local storage, streamed over a network, etc. A video encoding device may encode data and store it in storage, and / or a video decoding device may retrieve data from storage and decode it. In some instances, encoding and decoding are performed by devices that do not communicate with each other but only encode data to storage and / or retrieve data from storage and decode the data.

[0067] It should be understood that for each of the examples described above with reference to the video encoder 20, the video decoder 30 can be used to perform the reverse process. Regarding signaling syntax elements, the video decoder 30 can be used to receive and parse such syntax elements, and accordingly decode the associated video data. In some examples, the video encoder 20 can entropy encode one or more syntax elements defining… into an encoded video bitstream. In such instances, the video decoder 30 can parse such syntax elements and accordingly decode the associated video data.

[0068] Encoder & Encoding Method

[0069] Figure 2 A schematic block diagram illustrating an example of a video encoder 20 used to implement the technology disclosed in this application (disclosed). Figure 2In this example, the video encoder 20 includes a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a buffer 216, a loop filter unit 220, a decoded picture buffer (DPB) 230, a prediction processing unit 260, and an entropy coding unit 270. The prediction processing unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254, and a mode selection unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). Figure 2 The video encoder 20 shown can also be called a hybrid video encoder or a video encoder based on a hybrid video codec.

[0070] For example, the residual calculation unit 204, transform processing unit 206, quantization unit 208, prediction processing unit 260, and entropy coding unit 270 form the forward signal path of the encoder 20, while, for example, the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, buffer 216, loop filter 220, decoded picture buffer (DPB) 230, and prediction processing unit 260 form the backward signal path of the encoder, wherein the backward signal path of the encoder corresponds to the signal path of the decoder (see...). Figure 3 Decoder 30 in the middle.

[0071] Encoder 20 receives, for example, image 201 or block 203 of image 201 via input 202, such as images in a sequence of images forming a video or video sequence. Image block 203 may also be referred to as the current image block or the image block to be encoded, and image 201 may be referred to as the current image or the image to be encoded (especially in video encoding when the current image is distinguished from other images, such as previously encoded and / or decoded images in the same video sequence, i.e., the video sequence that also includes the current image).

[0072] segmentation

[0073] Embodiments of encoder 20 may include a segmentation unit ( Figure 2 (Not shown in the diagram) is used to segment image 201 into multiple blocks, such as block 203, typically into multiple non-overlapping blocks. The segmentation unit can be used to apply the same block size and corresponding raster defining the block size to all images in a video sequence, or to change the block size between images, subsets, or groups of images, and segment each image into corresponding blocks.

[0074] In one instance, the prediction processing unit 260 of the video encoder 20 can be used to perform any combination of the segmentation techniques described above.

[0075] As in image 201, block 203 is also, or can be considered as, a two-dimensional array or matrix of sampled points with luminance values ​​(sampled values), although its size is smaller than that of image 201. In other words, block 203 may include, for example, a single sampled array (e.g., a luminance array in the case of black and white image 201) or three sampled arrays (e.g., a luminance array and two chrominance arrays in the case of a color image) or any other number and / or type of array depending on the applied color format. The number of sampled points in the horizontal and vertical directions (or axes) of block 203 defines the size of block 203.

[0076] like Figure 2 The encoder 20 shown is used to encode the image 201 block by block, for example, to perform encoding and prediction for each block 203.

[0077] Residual calculation

[0078] The residual calculation unit 204 is used to calculate the residual block 205 based on the image block 203 and the prediction block 265 (other details of the prediction block 265 are provided below). For example, the residual block 205 is obtained in the sample domain by subtracting the sample value of the prediction block 265 from the sample value of the image block 203 on a sample-by-sample (pixel-by-pixel) basis.

[0079] Transformation

[0080] The transform processing unit 206 applies a transform, such as a discrete cosine transform (DCT) or a discrete sine transform (DST), to the sample values ​​of the residual block 205 to obtain transform coefficients 207 in the transform domain. The transform coefficients 207 can also be called transform residual coefficients and represent the residual block 205 in the transform domain.

[0081] Transform processing unit 206 can be used to apply integer approximations of DCT / DST, such as those specified for HEVC / H.265. These integer approximations are typically scaled by a factor compared to orthogonal DCT transforms. To maintain the norm of the residual blocks processed by the forward and inverse transforms, an additional scaling factor is applied as part of the transform process. The scaling factor is typically chosen based on certain constraints, such as a power of 2 for the shift operation, the bit depth of the transform coefficients, accuracy, and implementation cost. For example, a specific scaling factor can be specified on the decoder 30 side via, for example, inverse transform processing unit 212 (and on the encoder 20 side via, for example, inverse transform processing unit 212 for the corresponding inverse transform), and correspondingly, a corresponding scaling factor can be specified on the encoder 20 side via transform processing unit 206 for the forward transform.

[0082] Quantification

[0083] Quantization unit 208 is used to quantize the transform coefficients 207, for example, by applying scalar quantization or vector quantization, to obtain quantized transform coefficients 209. Quantized transform coefficients 209 can also be referred to as quantized residual coefficients 209. The quantization process can reduce the bit depth associated with some or all of the transform coefficients 207. For example, n-bit transform coefficients can be rounded down to m-bit transform coefficients during quantization, where n is greater than m. The degree of quantization can be modified by adjusting the quantization parameter (QP). For example, for scalar quantization, different scales can be applied to achieve finer or coarser quantization. Smaller quantization steps correspond to finer quantization, while larger quantization steps correspond to coarser quantization. The appropriate quantization step size can be indicated by the quantization parameter (QP). For example, the quantization parameter can be an index of a predefined set of appropriate quantization steps. For example, a smaller quantization parameter can correspond to fine quantization (smaller quantization step size), a larger quantization parameter can correspond to coarse quantization (larger quantization step size), and vice versa. Quantization can involve division by the quantization step size and a corresponding quantization or inverse quantization, performed, for example, by inverse quantization 210, or it can involve multiplication by the quantization step size. The quantization step size can be determined using quantization parameters according to some standard embodiments, such as HEVC. Generally, the quantization step size can be calculated based on the quantization parameters using a fixed-point approximation of an equation involving division. Additional scaling factors can be introduced for quantization and inverse quantization to recover the norm of the residual block, which may have been modified due to the scaling used in the fixed-point approximation of the equations used for the quantization step size and quantization parameters. In one example implementation, the scaling of the inverse transform and inverse quantization can be combined. Alternatively, a custom quantization table can be used and transmitted from the encoder to the decoder via a signal, for example, in a bitstream. Quantization is a lossy operation, where the loss increases with the quantization step size.

[0084] The inverse quantization unit 210 is used to apply the inverse quantization of the quantization unit 208 to the quantized coefficients to obtain the dequantized coefficients 211. For example, it applies the inverse quantization scheme of the quantization scheme applied by the quantization unit 208 based on or using the same quantization step size as the quantization unit 208. The dequantized coefficients 211 can also be called the dequantized residual coefficients 211, corresponding to the transform coefficients 207, although the loss due to quantization is usually different from that of the transform coefficients.

[0085] The inverse transform processing unit 212 is used to apply the inverse transform of the transform applied by the transform processing unit 206, such as the inverse discrete cosine transform (DCT) or the inverse discrete sine transform (DST), to obtain the inverse transform block 213 in the sample domain. The inverse transform block 213 can also be referred to as the inverse transform dequantized block 213 or the inverse transform residual block 213.

[0086] Reconstruction unit 214 (e.g., summer 214) is used to add inverse transform block 213 (i.e. reconstructed residual block 213) to prediction block 265 to obtain reconstructed block 215 in the sample domain, for example, by adding the sample value of reconstructed residual block 213 to the sample value of prediction block 265.

[0087] Optionally, for example, buffer unit 216 (or simply "buffer" 216) of line buffer 216 is used to buffer or store reconstructed block 215 and corresponding sample values ​​for, for example, intra-frame prediction. In other embodiments, the encoder can be used to perform any type of estimation and / or prediction, such as intra-frame prediction, using the unfiltered reconstructed block and / or corresponding sample values ​​stored in buffer unit 216.

[0088] For example, embodiments of encoder 20 can be configured such that buffer unit 216 is used not only to store the reconstructed block 215 for intra-frame prediction 254, but also for loop filter unit 220 (in Figure 2 (not shown in the image), and / or, for example, such that buffer unit 216 and decoded image buffer unit 230 form a buffer. Other embodiments may be used to filter block 221 and / or blocks or samples from decoded image buffer 230 (not shown in the image), and / or, for example, such that buffer unit 216 and decoded image buffer unit 230 form a buffer. Figure 2 (Not shown in the image) is used as input or basis for intra-frame prediction 254.

[0089] Loop filter unit 220 (or simply "loop filter" 220) is used to filter the reconstructed block 215 to obtain the filtered block 221, thereby facilitating pixel transformation or improving video quality. Loop filter unit 220 is intended to represent one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or other filters such as a bilateral filter, an adaptive loop filter (ALF), or a sharpening or smoothing filter, or a co-filter. Although loop filter unit 220 is used in... Figure 2The image shown is an in-loop filter, but in other configurations, the loop filter unit 220 may be implemented as a post-loop filter. The filtered block 221 may also be referred to as the filtered reconstructed block 221. The decoded image buffer 230 can store the reconstructed encoded block after the loop filter unit 220 performs the filtering operation on the reconstructed encoded block.

[0090] An embodiment of encoder 20 (correspondingly, loop filter unit 220) can be used to output loop filter parameters (e.g., sample adaptive offset information), for example, directly output or entropy encoded by entropy coding unit 270 or any other entropy coding unit, such that decoder 30 can receive and apply the same loop filter parameters for decoding.

[0091] The decoded picture buffer (DPB) 230 can be a reference picture memory for storing reference picture data for the video encoder 20 to encode video data. The DPB 230 can be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM) (including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM)) or other types of memory devices. The DPB 230 and buffer 216 can be provided by the same memory device or separate memory devices. In one example, the decoded picture buffer (DPB) 230 is used to store filtered block 221. The decoded picture buffer 230 can be further used to store other previously filtered blocks of the same current picture or different pictures, such as previously reconstructed pictures, for example, previously reconstructed and filtered blocks 221, and can provide complete previously reconstructed, i.e., decoded pictures (and corresponding reference blocks and samples) and / or partially reconstructed current pictures (and corresponding reference blocks and samples), for example, for inter-frame prediction. In one instance, if reconstructed block 215 can be reconstructed without in-loop filtering, the decoded picture buffer (DPB) 230 is used to store reconstructed block 215.

[0092] The prediction processing unit 260, also known as the block prediction processing unit 260, is used to receive or acquire block 203 (current block 203 of current image 201) and reconstructed image data, such as reference samples of the same (current) image from buffer 216 and / or reference image data 231 of one or more previously decoded images from decoded image buffer 230, and to process such data to make predictions, i.e., to provide prediction block 265 which can be inter-frame prediction block 245 or intra-frame prediction block 255.

[0093] The mode selection unit 262 can be used to select a prediction mode (e.g., intra-frame or inter-frame prediction mode) and / or the corresponding prediction block 245 or 255 used as prediction block 265 to compute residual block 205 and reconstructed block 215.

[0094] Embodiments of the mode selection unit 262 can be used to select a prediction mode (e.g., from those prediction modes supported by the prediction processing unit 260), which provides the best match or minimum residual (minimum residual means better compression in transmission or storage), or provides minimum signaling overhead (minimum signaling overhead means better compression in transmission or storage), or considers or balances both. The mode selection unit 262 can be used to determine the prediction mode based on rate distortion optimization (RDO), i.e., selecting the prediction mode that provides minimum RDO optimization, or selecting the prediction mode whose relevant rate distortion at least meets the prediction mode selection criteria.

[0095] The following will explain in detail the prediction processing performed by instances of encoder 20 (e.g., by prediction processing unit 260) and the mode selection performed (e.g., by mode selection unit 262).

[0096] As described above, encoder 20 is used to determine or select the best or optimal prediction mode from a (predetermined) set of prediction modes. The set of prediction modes may include, for example, intra-frame prediction modes and / or inter-frame prediction modes.

[0097] The intra-prediction mode set can include 35 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in H.265, or it can include 67 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in the developing H.266.

[0098] The set of (possible) inter-frame prediction modes depends on the available reference image (i.e., at least a partially decoded image stored in the DBP 230 as described above) and other inter-frame prediction parameters, such as whether to use the entire reference image or only a portion of the reference image, such as a search window region surrounding the current block, to search for the best matching reference block, and / or whether to apply pixel interpolation such as half-pixel and / or quarter-pixel interpolation.

[0099] In addition to the prediction modes mentioned above, skip mode and / or direct mode can also be applied.

[0100] The prediction processing unit 260 can be further used to divide block 203 into smaller block partitions or sub-blocks, for example, by iteratively using quad-tree (QT) partitioning, binary-tree (BT) partitioning, or triple-tree (TT) partitioning, or any combination thereof, and to perform prediction for each of the block partitions or sub-blocks, for example, wherein mode selection includes selecting the tree structure of the partitioned block 203 and selecting a prediction mode to be applied to each of the block partitions or sub-blocks.

[0101] Inter-frame prediction unit 244 may include motion estimation (ME) units. Figure 2 (not shown in the image) and motion compensation (MC) unit ( Figure 2 (Not shown in the image). The motion estimation unit is used to receive or acquire image block 203 (current image block 203 of current image 201) and decoded image 231, or at least one or more previously reconstructed blocks, such as reconstructed blocks of one or more other / different previously decoded images 231, to perform motion estimation. For example, the video sequence may include the current image and the previously decoded image 31, or in other words, the current image and the previously decoded image 31 may be part of the image sequence that forms the video sequence, or form the image sequence.

[0102] For example, encoder 20 can be used to select a reference block from multiple reference blocks of the same or different images in multiple other images, and send it to the motion estimation unit ( Figure 2 (Not shown in the image) Provides a reference image (or reference image index...) and / or provides the offset (spatial offset) between the position (X, Y coordinates) of the reference block and the position of the current block as an inter-frame prediction parameter. This offset is also called a motion vector (MV).

[0103] The motion compensation unit is used to acquire, for example, receive inter-frame prediction parameters, and perform inter-frame prediction based on or using the inter-frame prediction parameters to acquire inter-frame prediction block 245. This is achieved by the motion compensation unit ( Figure 2 Motion compensation performed (not shown) may involve extracting or generating prediction blocks based on motion / block vectors determined through motion estimation (possibly performing interpolation with subpixel accuracy). Interpolation filtering can generate additional pixel samples from known pixel samples, potentially increasing the number of candidate prediction blocks available for encoding picture blocks. Once the motion vector of the PU for the current picture block is received, the motion compensation unit 246 can locate the prediction block pointed to by the motion vector in a list of reference pictures. The motion compensation unit 246 can also generate syntax elements associated with blocks and video stripes for use by the video decoder 30 when decoding picture blocks of the video stripe.

[0104] Intra-prediction unit 254 is used to acquire, for example, image block 203 (current image block) of the same image and one or more previously reconstructed blocks, such as reconstructed adjacent blocks, for intra-prediction. For example, encoder 20 can be used to select an intra-prediction mode from a plurality of (predetermined) intra-prediction modes.

[0105] An embodiment of encoder 20 can be used to select an intra-prediction mode based on optimization criteria, such as based on minimum residual (e.g., providing an intra-prediction mode that most closely resembles the current image block 203) or minimum bitrate distortion.

[0106] Intra-prediction unit 254 is further configured to determine intra-prediction block 255 based on intra-prediction parameters as selected for the intra-prediction mode. In any case, after selecting the intra-prediction mode for the block, intra-prediction unit 254 is also configured to provide intra-prediction parameters to entropy coding unit 270, i.e., to provide information indicating the selected intra-prediction mode for the block. In one instance, intra-prediction unit 254 may be configured to perform any combination of intra-prediction techniques described below.

[0107] Entropy coding unit 270 is used to apply an entropy coding algorithm or scheme (e.g., variable length coding (VLC), context adaptive VLC (CAVLC), arithmetic coding, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to one or all (or not to apply) of the quantized residual coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, and / or loop filter parameters to obtain encoded picture data 21 that can be output as, for example, an encoded bitstream 21, through output 272. The encoded bitstream can be transmitted to video decoder 30, or archived for later transmission or retrieval by video decoder 30. Entropy coding unit 270 can also be used to entropy code other syntax elements of the current video strip being encoded.

[0108] Other structural variations of the video encoder 20 can be used to encode video streams. For example, a non-transform-based encoder 20 can directly quantize the residual signal without a transform processing unit 206 for certain blocks or frames. In another embodiment, the encoder 20 may have a quantization unit 208 and an inverse quantization unit 210 combined into a single unit.

[0109] Figure 3 An exemplary video decoder 30 is illustrated for implementing the techniques of this application, namely the image prediction method. The video decoder 30 is used to receive, for example, encoded image data (e.g., encoded bitstream) 21 encoded by encoder 20, to obtain a decoded image 231. During the decoding process, the video decoder 30 receives video data from the video encoder 20, such as encoded video bitstreams representing image blocks of encoded video stripes and associated syntax elements.

[0110] exist Figure 3In one example, decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), a buffer 316, a loop filter 320, a decoded image buffer 330, and a prediction processing unit 360. Prediction processing unit 360 may include an inter-frame prediction unit 344, an intra-frame prediction unit 354, and a mode selection unit 362. In some instances, video decoder 30 can perform substantially the same functions as the referenced unit. Figure 2 The video encoder 20 describes the encoding passes as the inverse of the decoding passes.

[0111] Entropy decoding unit 304 is used to perform entropy decoding on encoded image data 21 to obtain, for example, quantization coefficients 309 and / or decoded encoded parameters. Figure 3 (Not shown in the image), for example, any one or all of the (decoded) inter-frame prediction, intra-frame prediction parameters, loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 is further configured to forward the inter-frame prediction parameters, intra-frame prediction parameters, and / or other syntax elements to the prediction processing unit 360. The video decoder 30 may receive syntax elements at the video strip level and / or video block level.

[0112] The inverse quantization unit 310 can function the same as the inverse quantization unit 110, the inverse transform processing unit 312 can function the same as the inverse transform processing unit 212, the reconstruction unit 314 can function the same as the reconstruction unit 214, the buffer 316 can function the same as the buffer 216, the loop filter 320 can function the same as the loop filter 220, and the decoded image buffer 330 can function the same as the decoded image buffer 230.

[0113] The prediction processing unit 360 may include an inter-frame prediction unit 344 and an intra-frame prediction unit 354, wherein the inter-frame prediction unit 344 may be functionally similar to the inter-frame prediction unit 244, and the intra-frame prediction unit 354 may be functionally similar to the intra-frame prediction unit 254. The prediction processing unit 360 is typically used to perform block prediction and / or obtain prediction blocks 365 from encoded data 21, and to receive or obtain prediction-related parameters and / or information about the selected prediction mode from, for example, an entropy decoding unit 304 (explicitly or implicitly).

[0114] When a video strip is encoded as an intra-coded (I) strip, the intra-prediction unit 354 of the prediction processing unit 360 generates a prediction block 365 for a picture block of the current video strip based on the intra-prediction mode of the signal representation and data from previously decoded blocks of the current frame or picture. When a video frame is encoded as an inter-coded (i.e., B or P) strip, the inter-prediction unit 344 (e.g., a motion compensation unit) of the prediction processing unit 360 generates a prediction block 365 for a video block of the current video strip based on motion vectors and other syntax elements received from the entropy decoding unit 304. For inter-frame prediction, a prediction block can be generated from a reference picture within a list of reference pictures. The video decoder 30 can construct a list of reference frames, list 0 and list 1, based on reference pictures stored in the DPB 330 using a default construction technique.

[0115] The prediction processing unit 360 is used to determine prediction information for video blocks of the current video strip by parsing motion vectors and other syntax elements, and to generate prediction blocks for the current video blocks for proper decoding using the prediction information. For example, the prediction processing unit 360 uses some received syntax elements to determine the prediction mode (e.g., intra-frame or inter-frame prediction), inter-frame prediction stripe type (e.g., B-strip, P-strip, or GPB-strip), construction information of one or more reference picture lists for the stripe, motion vectors for each inter-frame coded video block of the stripe, inter-frame prediction state of each inter-frame coded video block of the stripe, and other information to decode the video blocks of the current video stripe.

[0116] The inverse quantization unit 310 can be used to inverse quantize (i.e., dequantize) the quantized transform coefficients provided in the bitstream and decoded by the entropy decoding unit 304. The inverse quantization process may involve using quantization parameters calculated by the video encoder 20 for each video block in the video strip to determine the degree of quantization to be applied and similarly determine the degree of inverse quantization to be applied.

[0117] The inverse transform processing unit 312 is used to apply an inverse transform (e.g., inverse DCT, inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients in order to generate a residual block in the pixel domain.

[0118] Reconstruction unit 314 (e.g., summer 314) is used to add inverse transform block 313 (i.e. reconstructed residual block 313) to prediction block 365 to obtain reconstructed block 315 in the sample domain, for example by adding the sample values ​​of reconstructed residual block 313 to the sample values ​​of prediction block 365.

[0119] Loop filter unit 320 (during or after the encoding loop) is used to filter the reconstructed block 315 to obtain the filtered block 321, thereby facilitating pixel transformation or improving video quality. In one instance, loop filter unit 320 can be used to perform any combination of the filtering techniques described below. Loop filter unit 320 is intended to represent one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or other filters such as a bilateral filter, an adaptive loop filter (ALF), or a sharpening or smoothing filter, or a co-filter. Although loop filter unit 320 is used in... Figure 3 The loop filter unit 320 is shown as an in-loop filter, but in other configurations, it can be implemented as a post-loop filter.

[0120] The decoded video block 321 in the given frame or image is then stored in the decoded image buffer 330, which stores a reference image for subsequent motion compensation.

[0121] Decoder 30 is used, for example, to output decoded image 31 via output 332 for presentation to or viewing by the user.

[0122] Other variations of the video decoder 30 can be used to decode compressed bitstreams. For example, the decoder 30 can generate an output video stream without the loop filter unit 320. For example, the non-transform-based decoder 30 can directly inverse quantize the residual signal without the inverse transform processing unit 312 for certain blocks or frames. In another embodiment, the video decoder 30 may have an inverse quantization unit 310 and an inverse transform processing unit 312 combined into a single unit.

[0123] Figure 4 It is a inclusion according to an exemplary embodiment Figure 2 encoder 20 and / or Figure 3 The illustrated diagram shows an example of a video encoding system 40 with decoder 30. System 40 can implement various combinations of technologies of this application. In the illustrated embodiment, video encoding system 40 may include imaging device 41, video encoder 20, video decoder 30 (and / or video encoder implemented by logic circuitry 47 of processing unit 46), antenna 42, one or more processors 43, one or more memories 44 and / or display device 45.

[0124] As shown in the figure, the imaging device 41, antenna 42, processing unit 46, logic circuit 47, video encoder 20, video decoder 30, processor 43, memory 44, and / or display device 45 are capable of communicating with each other. As discussed, although the video encoding system 40 is illustrated with video encoder 20 and video decoder 30, in different instances, the video encoding system 40 may contain only video encoder 20 or only video decoder 30.

[0125] In some instances, as shown, the video encoding system 40 may include an antenna 42. For example, the antenna 42 may be used to transmit or receive encoded bitstreams of video data. Additionally, in some instances, the video encoding system 40 may include a display device 45. The display device 45 may be used to present the video data. In some instances, as shown, the logic circuitry 47 may be implemented using a processing unit 46. The processing unit 46 may include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, etc. The video encoding system 40 may also include an optional processor 43, which may similarly include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, etc. In some instances, the logic circuitry 47 may be implemented using hardware, such as dedicated video encoding hardware, while the processor 43 may be implemented using general-purpose software, an operating system, etc. Additionally, memory 44 can be any type of memory, such as volatile memory (e.g., Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.). In a non-limiting example, memory 44 can be implemented as a high-speed cache memory. In some examples, logic circuitry 47 can access memory 44 (e.g., for implementing an image buffer). In other examples, logic circuitry 47 and / or processing unit 46 can include memory (e.g., cache, etc.) for implementing image buffers, etc.

[0126] In some instances, the video encoder 20 implemented via logic circuitry may include (e.g., implemented via processing unit 46 or memory 44) an image buffer and (e.g., implemented via processing unit 46) a graphics processing unit. The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video encoder 20 implemented via logic circuitry 47 to implement reference... Figure 2And / or any other encoder system or subsystem described herein, and the various modules discussed herein. Logic circuits may be used to perform the various operations discussed herein.

[0127] The video decoder 30 can be implemented in a similar manner via logic circuit 47 to implement the reference. Figure 3 The decoder 30 and / or any other decoder system or subsystem described herein are various modules discussed. In some instances, the logic circuit-implemented video decoder 30 may include an image buffer (implemented via processing unit 2820 or memory 44) and a graphics processing unit (e.g., implemented via processing unit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video decoder 30 implemented via logic circuit 47 to implement reference Figure 3 And / or the various modules discussed in any other decoder system or subsystem described herein.

[0128] In some instances, the antenna 42 of the video coding system 40 can be used to receive a encoded bitstream of video data. As discussed herein, the encoded bitstream may contain data related to encoded video frames, indicators, index values, mode selection data, etc., such as data related to code segmentation (e.g., transform coefficients or quantized transform coefficients, optional indicators, and / or data defining code segmentation). The video coding system 40 may also include a video decoder 30 coupled to the antenna 42 for decoding the encoded bitstream. A display device 45 is used to present the video frames.

[0129] Figure 5 It is available as an exemplary embodiment. Figure 1 A simplified block diagram of device 500 comprising one or both of the source device 12 and the destination device 14. Device 500 can implement the technology of this application for image prediction. Device 500 can take the form of a computing system comprising multiple computing devices, or take the form of a single computing device such as a mobile phone, tablet computer, laptop computer, notebook computer, or desktop computer.

[0130] The processor 502 in device 500 can be a central processing unit. Alternatively, processor 502 can be any other type of device or multiple devices, existing or to be developed in the future, capable of manipulating or processing information. As shown, while the disclosed implementation can be practiced using a single processor, such as processor 502, advantages in speed and efficiency can be achieved by using more than one processor.

[0131] In one embodiment, the memory 504 in device 500 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory 504. Memory 504 may include code and data 506 accessed by processor 502 using bus 512. Memory 504 may further include an operating system 508 and an application 510, which contains at least one program that permits processor 502 to perform the methods described herein. For example, application 510 may include applications 1 to N, which further include video encoding applications that perform the methods described herein. Device 500 may also include additional memory in the form of a slave memory 514, which may be, for example, a memory card used with a mobile computing device. Because video communication sessions may contain a large amount of information, this information may be stored wholly or partially in slave memory 514 and loaded into memory 504 as needed for processing.

[0132] Device 500 may also include one or more output devices, such as display 518. In one example, display 518 may be a touch-sensitive display that combines a display with a touch-sensitive element operable to sense touch input. Display 518 may be coupled to processor 502 via bus 512. In addition to display 518, other output devices may be provided that allow a user to program or otherwise use device 500, or other output devices may be provided as alternatives to display 518. When the output device is a display or includes a display, the display may be implemented in various ways, including via a liquid crystal display (LCD), a cathode ray tube (CRT) display, a plasma display, or a light-emitting diode (LED) display, such as an organic LED (OLED) display.

[0133] The device 500 may also include or be connected to an image sensing device 520, such as a camera or any other existing or future image sensing device capable of sensing images, such as images of a user operating the device 500. The image sensing device 520 may be positioned directly facing the user operating the device 500. In one example, the position and optical axis of the image sensing device 520 may be configured such that its field of view includes an area immediately adjacent to and visible from the display 518.

[0134] The device 500 may also include or be connected to a sound sensing device 522, such as a microphone or any other existing or future sound sensing device capable of sensing sounds near the device 500. The sound sensing device 522 may be positioned directly facing the user operating the device 500 and may be used to receive sounds emitted by the user while operating the device 500, such as speech or other sounds.

[0135] Although Figure 5 The processor 502 and memory 504 of device 500 are illustrated as being integrated into a single unit, but other configurations may also be used. The operation of processor 502 may be distributed across multiple directly coupled machines (each machine having one or more processors), or distributed across a local area or other network. Memory 504 may be distributed across multiple machines, such as network-based memory or memory in multiple machines running device 500. Although only a single bus is illustrated here, bus 512 of device 500 may be formed by multiple buses. Furthermore, memory 514 may be directly coupled to other components of device 500 or accessible via a network, and may include a single integrated unit, such as a memory card, or multiple units, such as multiple memory cards. Therefore, device 500 can be implemented in a variety of configurations. The terminology used in this application is briefly described below:

[0136] Intra-frame prediction coding: A coding method that uses the values ​​of surrounding neighboring pixels to predict the current pixel value, and then encodes the prediction error.

[0137] Encoded image: The encoded representation of an image containing all the coding tree units of the image.

[0138] Motion vector (MV): A two-dimensional vector used for inter-frame prediction that provides the offset from coordinates in the decoded image to coordinates in the reference image.

[0139] Prediction block: A rectangular M×N sample block on which the same prediction is applied.

[0140] Prediction process: Use the predicted values ​​to provide an estimate of the currently decoded data element (e.g., sample value or motion vector).

[0141] Predicted value: A combination of previously decoded data elements (e.g., sample values ​​or motion vectors) used in the decoding process of a specified value or subsequent data elements.

[0142] Reference frame: An image or frame that serves as a short-term or long-term reference image. Reference frames contain samples that can be used for inter-frame prediction during the decoding process of subsequent images, in the order they are decoded.

[0143] Inter-frame prediction: Based on the pixels in the reference frame of the current block, the predicted image of the current block is generated by using motion vectors to indicate the positions of the pixels used for prediction in the reference frame.

[0144] Bidirectional prediction (B) slice: A slice that can be decoded by predicting sample values ​​for each block using up to two motion vectors and a reference index using either intra-frame prediction or inter-frame prediction.

[0145] CTU: Coding Tree Unit. An image is composed of multiple CTUs. A CTU typically corresponds to a square image region, containing the luminance and chrominance pixels in that region (or it may contain only luminance pixels, or only chrominance pixels). A CTU also contains syntax elements that indicate how to divide the CTU into at least one coding unit (CU), and the method for decoding each coding unit to obtain the reconstructed image.

[0146] CU: Coding Unit, corresponding to an A×B rectangular region in the image, containing A×B luminance pixels and / or its corresponding chrominance pixels. A is the width of the rectangle, and B is the height. A and B can be the same or different, and their values ​​are usually integer powers of 2, such as 128, 64, 32, 16, 8, and 4. A coding unit contains a prediction image and a residual image. The prediction image and the residual image are added together to obtain the reconstructed image of the coding unit. The prediction image is generated through intra-frame prediction or inter-frame prediction, and the residual image is generated by inverse quantization and inverse transform processing of the transform coefficients.

[0147] VTM: A new codec reference software developed by the JVET organization.

[0148] Merge coding: an inter-frame coding method in which motion vectors are not directly transmitted in the bitstream. The current block can select a corresponding merging candidate from the merge candidate list based on the merge index, and use the motion information of the merging candidate as the motion information of the current block, or scale the motion information of the merging candidate as the motion information of the current block.

[0149] The image prediction method described in this application is related to inter-frame prediction, and the multiple hypothesis method is one such inter-frame prediction method. The following is a brief description of inter-frame prediction, which is mainly used in video encoding and decoding to eliminate temporal and spatial redundancy in video.

[0150] Inter-frame prediction is a prediction technique based on motion compensation. In inter-frame predictive coding, since there is a certain temporal correlation between the same objects in neighboring frames of an image, each frame of the image sequence can be divided into many non-overlapping blocks, and it is assumed that all pixels within a block have the same motion. The main processing involves determining the motion information of the current block, obtaining a reference image block from the reference frame based on the motion information, and generating the predicted image of the current block. The motion information includes the inter-frame prediction direction, the reference frame index (ref_idx), and the motion vector (MV). Inter-frame prediction uses the inter-frame prediction direction to indicate which prediction direction the current block should use: forward prediction, backward prediction, or bidirectional prediction. The reference frame index identifies the reference frame, and the motion vector indicates the position offset of the reference block in the reference frame relative to the current block in the current frame. The motion vector indicates the displacement vector of the reference image block used to predict the current block in the reference frame relative to the current block; therefore, one motion vector corresponds to one reference image block.

[0151] During encoding, video coding standards such as H.265 / HEVC and H.266 / VVC divide a frame of image into non-overlapping Coding Tree Units (CTUs), and a CTU is further divided into one or more Coding Units (CUs). A CU contains coding information, including prediction modes and transform coefficients. At the decoding end, the CUs are processed according to this coding information, including prediction, inverse quantization, and inverse transform, to generate the reconstructed image corresponding to that CU.

[0152] Motion information occupies a large portion of the data in the bitstream. To reduce the required data volume, motion information is typically transmitted using prediction. Generally, this can be divided into two modes: inter and merge.

[0153] Inter MVP mode: The transmitted motion information includes: inter-frame prediction direction (forward, backward, or bidirectional), reference frame index, motion vector prediction index, and motion vector residual value. For motion vector information, the difference between the actual motion vector and the motion vector predictor (MVP) is typically transmitted. The encoder passes the motion vector difference (MVD) between the MVP and the actual motion vector to the decoder. The motion vector prediction may contain multiple prediction values. Generally, the same method is used in both the encoder and decoder to construct the motion vector prediction candidate list (MVP candidate list), and the motion vector predictor index (MVP index) is passed to the decoder.

[0154] Merge mode: A merge candidate list is constructed using the same method in both the encoding and decoding segments, and the index is passed to the decoding end. The merge index is transmitted in the bitstream. The motion information in the candidate list is typically obtained from its spatially adjacent blocks or temporally adjacent blocks in the reference frame. The candidate motion information obtained from the motion information of the image blocks adjacent to the current block is called a spatial candidate, and the motion information of the corresponding image block obtained from the current block in the reference image is called a temporal candidate.

[0155] Bidirectional optical flow (BIO) is an inter-frame coding method that improves pixel-level motion vector optimization based on block-based motion compensation techniques (related technical proposals include JVET-E1001, JVET-K0255, and JVET-K0119). BIO provides improved motion vectors at the pixel level without requiring additional cumbersome searches or extra bitstream information. For each pixel's motion trajectory, BIO combines optical flow field and Hermite interpolation analysis to obtain the improved motion vector by minimizing the intersection of the motion trajectory and the reference frame plane. This process utilizes the horizontal and vertical gradients of each pixel in the prediction block, and a special 6-tap filter is used for gradient interpolation.

[0156] Decoder-side motion vector refinement (DMVR) is an inter-frame coding method that adjusts existing motion vectors at the decoder through a search (related technical proposals include JVET-E1001 and JVET-K0275). DMVR is used on coded blocks that utilize bidirectional patterns for inter-frame prediction. At the decoder, a bidirectional template is first obtained by generating two prediction blocks using motion compensation from the motion vectors MV0 (list0) and MV1 (list1) transmitted in the initial bitstream. The bidirectional template is then used to search around the positions indicated by the initial motion vectors MV0 and MV1 in the reference frames of list0 and list1, finding the positions with the minimum matching distortion within the search range as new MV0 and MV1, along with two new prediction blocks. The final prediction block is then generated by weighting the new prediction blocks.

[0157] In HEVC and earlier standards, reference frames are divided into two groups: forward and backward, placed in two reference picture lists, typically named list0 and list1. The inter-frame prediction direction indicates whether the current block should use forward, backward, or bidirectional prediction. Different reference picture lists (list0, list1, or both) are selected based on the prediction direction. For each selected reference picture list, the reference frame is identified by its index. Within the selected reference frame, motion vectors indicate the offset of the current block's prediction block relative to the current block in the current frame. Then, based on the prediction direction, the final prediction block is generated using prediction blocks obtained from reference frames in list0, list1, or both. Specifically, when the prediction direction is unidirectional, prediction blocks from reference frames in list0 or list1 are used directly; when the prediction direction is bidirectional, prediction blocks from reference frames in list0 and list1 are synthesized using a weighted average. In multi-hypothesis inter-prediction (MPI) methods, the predicted block or the final predicted block is used as the original hypothesis for the current block. New motion or prediction information is added to increase the number of hypotheses, thereby improving coding performance. JVET-K0269 and JVET-K0257 proposals use MPI methods, but they implement them in different ways.

[0158] The multi-hypothesis coding method in JVET-K0269 adds additional hypotheses by transmitting motion information, building upon existing inter-frame coding in HEVC or VVC VTM. Each additional hypothesis requires transmitting an additional hypothesis flag, a reference frame number (ref idx add hyp), motion vector prediction information (mvp add hyp flag), motion vector difference values, and a weighting factor (add hyp weight idx). The meanings of the reference frame number, motion vector prediction information, and motion vector difference values ​​are the same as in HEVC and previous standards, but the generation method of the reference frame list differs. The reference frame list for additional hypotheses is generated by alternating between list0 and list1 in HEVC or VVC VTM, with duplicate reference frames not added. After obtaining the prediction block (additional hypothesis) using the motion information of the additional hypotheses, it is compared with the original final prediction block (original hypothesis) using the weighting factor of the additional hypothesis. There are two weighting factors for the additional hypotheses: 1 / 4 and -1 / 8.

[0159] The image prediction method described in this application is used to predict at least one image patch in an image using inter-frame prediction, thereby obtaining a predicted image patch. Subsequently, further decoding processing can be performed based on the obtained predicted image patch to obtain a reconstructed image of the image patch. The processing at the encoding and decoding ends is the same. The following description uses the decoding end as an example. For the implementation of the encoding end, please refer to the encoding flow described above and the decoding method flow described below. For example, the encoding end obtains the multi-hypothesis coding information of the image patch to be encoded from the multi-hypothesis coding information list by traversing it. This multi-hypothesis coding information includes an identifier representing the partitioning method of the multi-hypothesis coding method, and parameters of the multi-hypothesis coding method. This partitioning method is used when dividing the predicted blocks of multiple hypotheses of an image to be encoded using the multi-hypothesis coding method. For example, if the partitioning method is triangular partitioning, the parameter is the direction of the triangular partitioning; if the partitioning method is square partitioning, the parameter is the weighting coefficient.

[0160] Triangular partitioning can refer to the multi-hypothesis coding method in JVET-K0144 proposal. Based on the inter-frame coding in existing HEVC or VVCVTM, when the current coding block is merged or skipped, the coding block is divided into two triangular coding blocks diagonally. Each triangular coding block uses separate motion information obtained from the fusion motion information candidate list. The partitioning direction of the two triangular coding blocks and the combination of selected fusion motion information candidates are selected using a combination number from a preset multi-hypothesis fusion motion information candidate selection and partitioning combination list, and this number is transmitted in the bitstream. The two triangular coding blocks have a certain overlap area, and a distance-related weighting coefficient is used for weighting.

[0161] The square partitioning method can refer to the JVET-K0257 proposal, which uses a combined coding scheme to implement multi-hypothesis coding, combining merge with inter-MVP, merge, and intra-frame coding. When the current original hypothesis / first hypothesis is in inter-MVP mode and a unidirectional approach (i.e., only list0 or list1) is used, the merge mode is used to generate additional hypotheses. When the current original hypothesis / first hypothesis is in merge mode, the next fusion motion information candidate selected from the fusion motion information candidate list of the original hypothesis / first hypothesis can be used as the motion information for generating additional hypotheses. When the current original hypothesis / first hypothesis is in merge mode, intra-frame coding can also be used to generate additional hypotheses, with additional syntax elements indicating whether this method is used. After obtaining the prediction block (additional hypothesis), weighted coefficients are used to combine it with the original final prediction block. In the JVET-K0257 proposal, the weighted coefficients and the method are related to the combination of multiple hypotheses.

[0162] The process of generating the reference frame list is existing technology. This process can be performed using the same method as HEVC (i.e., constructing list0 and list1 at the beginning of the SLICE), or it can use other methods for generating the reference frame list (such as the method in the JVET-K0269 proposal). This invention is not limited to these methods. To improve efficiency, multi-hypothesis coding methods (such as the JVET-K0269 and JVET-K0257 proposals) add additional predictions besides the original set of forward, backward, or bidirectional predictions (the first hypothesis). In this invention, these are referred to as additional hypotheses, which may include the second hypothesis, the third hypothesis, etc. In this invention, the original set of forward, backward, or bidirectional predictions is referred to as the original hypothesis or the first hypothesis.

[0163] The following is combined Figure 6 This application describes an image prediction method. The image prediction method includes: 602: parsing the bitstream to obtain the multi-hypothesis information index of the currently to-be-decoded image block.

[0164] 602 corresponds to step 1 mentioned below. The multi-hypothesis information index is the multi-hypothesis pattern information mentioned below.

[0165] 604: Based on the multi-hypothesis information index, the first multi-hypothesis information corresponding to the current image block is obtained from the multi-hypothesis information list. The first multi-hypothesis information includes a motion information index and a first identifier. The motion information index of the first multi-hypothesis information indicates the multi-hypothesis motion information of the current image block to be decoded. The first identifier indicates the first partitioning method of the multi-hypothesis method. The first multi-hypothesis information also includes the parameters of the first partitioning method.

[0166] In one possible implementation, the list of multiple hypothesis information includes at least one first multiple hypothesis information and at least one second multiple hypothesis information. The second multiple hypothesis information includes a motion information index and a second identifier. The motion information index of the second multiple hypothesis information indicates the multiple hypothesis motion information of an image patch, and the second identifier indicates a second partitioning method of the multiple hypothesis method. The second multiple hypothesis information also includes parameters of the second partitioning method.

[0167] In one possible implementation, the first identifier and the second identifier are different values ​​of the same flag bit.

[0168] In one possible implementation, the first partitioning method is a triangular partition, where the parameters of the first partitioning method indicate the partitioning direction of the triangle; the second partitioning method is a square partition, where the parameters of the second partitioning method indicate the weighting coefficients of the square partition. Alternatively, the second partitioning method could be a triangular partition, where the parameters of the second partitioning method indicate the partitioning direction of the triangle; and the first partitioning method could be a square partition, where the parameters of the first partitioning method indicate the weighting coefficients of the square partition.

[0169] It should be understood that the first and second partitioning methods are different. Each partitioning method is used to divide the predicted image patches for multiple hypotheses, that is, to divide the predicted image patches for the original hypothesis and the predicted image patches for the additional hypotheses. For specific methods and approaches to partitioning the predicted image patches using the multiple hypotheses method, please refer to the content of JVET-K0144 proposal (triangular partitioning) and JVET-K0257 proposal (square partitioning).

[0170] In one possible implementation, the first partitioning method is a triangular partitioning, and the parameters of the first partitioning method indicate the partitioning direction of the triangular partitioning; or the first partitioning method is a square partitioning, and the parameters of the first partitioning method indicate the weighting coefficients of the square partitioning.

[0171] 606: Motion compensation is performed based on the motion information of the multi-hypothesis encoding of the current image block to be decoded, so as to obtain the predicted image block of multiple hypotheses.

[0172] 604 and 606 correspond to step 2 below.

[0173] 608: Based on the parameters of the first identifier and the first partitioning method, the first partitioning method is used to process the multiple hypothetical predicted image blocks to obtain the predicted image block of the current image block to be decoded.

[0174] 608 corresponds to step 3 below. The predicted image block of the current image block to be decoded can also be called the final inter-frame predicted image of the current image block to be decoded.

[0175] In one possible implementation, the method further includes: obtaining the multi-hypothesis motion information of the current image block to be decoded from the candidate motion information list of the current image block to be decoded based on the motion information index of the first multi-hypothesis information, wherein the multi-hypothesis motion information of the current image block to be decoded includes the motion information of the original hypothesis and the motion information of the additional hypothesis.

[0176] In one possible implementation, the motion information index of the first multiple hypothesis information includes a first index and a second index. The process of obtaining the multiple hypothesis motion information of the current image block to be decoded from the candidate motion information list of the current image block to be decoded based on the motion information index of the first multiple hypothesis information includes: obtaining the motion information of the original hypothesis from the first candidate motion information list of the current image block based on the first index, and obtaining the motion information of the additional hypothesis from the second candidate motion information list of the current image block based on the second index.

[0177] In this way, by using the above method, since the bitstream carries a partitioning identifier indicating the multi-hypothesis method, the decoding end can determine which partitioning method to use in the multi-hypothesis method to predict the image. That is, different partitioning methods can be used to obtain the prediction blocks of different image blocks. In this way, the image blocks can use a partitioning method that is more suitable for their own characteristics, thereby improving the efficiency of image prediction and reducing the decoding time.

[0178] The following describes the usage in detail. Figure 6 The corresponding image prediction method is used to obtain the decoded data block. Further implementation details of the above image prediction method can be found in the detailed description below. In the following process, the current block refers to the current image block described above.

[0179] The decoding process includes steps 1 through 4. The block that is currently undergoing decoding is called the current block.

[0180] Step 1: Parse the bitstream of the current block to obtain the prediction mode information of the current block.

[0181] Specifically, the mode information of the current block includes, but is not limited to, encoding information such as merge, skip, inter mode information, and multiple hypothesis mode information.

[0182] Step 2: Obtain the motion information of the current block based on the prediction mode information of the current block;

[0183] If the multi-hypothesis encoding mode is determined to be used in the current block based on the multi-hypothesis mode information, then the multi-hypothesis information of the current block is obtained.

[0184] Specifically, it includes the following:

[0185] Generate a candidate list for fused motion information. Specifically, this includes adding spatial and temporal candidates related to the current block to the candidate list for fused motion information of the current block, using the same method as in HEVC. For example... Figure 1 As shown, the spatial fusion candidates include A0, A1, B0, B1, and B2, while the temporal fusion candidates include T0 and T1. In VTM, the temporal fusion candidates also include candidates provided by the Adaptive Temporal Motion Vector Prediction (ATMVP) technique. This invention does not involve the process of generating the fused motion information candidate list, which can be performed using methods in HEVC or VTM, or other methods for generating the fused motion information candidate list, such as the method in JVET-K0257 proposal.

[0186] Then, the motion information of the two hypotheses and the multi-hypothesis motion compensation information are obtained. At the decoding end: based on the multi-hypothesis information index carried in the bitstream, the multi-hypothesis information corresponding to the current block is determined from the multi-hypothesis information list (hereinafter also referred to as the combination list of multi-hypothesis combination modes and motion information). Each multi-hypothesis information in the multi-hypothesis information list contains: a motion index, specifically the fusion motion information candidate list index of the first hypothesis (i.e., the original hypothesis mentioned above) and the fusion motion information candidate list index of the second hypothesis (i.e., the additional hypothesis mentioned above), a partitioning mode identifier (i.e., the first identifier or the second identifier mentioned above), and partitioning mode information (i.e., the parameters of the first partitioning method or the parameters of the second partitioning method mentioned above).

[0187] The motion information of the first hypothesis is confirmed from the fusion motion information candidate list based on the index of the first hypothesis in the multi-hypothesis information corresponding to the current block. The motion information of the second hypothesis is confirmed from the fusion motion information candidate list based on the index of the second hypothesis in the multi-hypothesis information corresponding to the current block. The partitioning mode identifier indicates whether the partitioning mode is triangular or square. When the partitioning mode identifier indicates a triangular partitioning mode, this partitioning mode information indicates the partitioning direction of the triangular partitioning mode; when the partitioning mode identifier indicates a square partitioning mode, this partitioning mode information indicates the weighting coefficients of the square partitioning.

[0188] The partitioning mode identifier is used to indicate whether the partitioning mode is triangular or square. When the partitioning mode identifier indicates that the partitioning mode is triangular, the partitioning mode information indicates the partitioning direction of the triangular partitioning mode; when the partitioning mode identifier indicates that the partitioning mode is square, the partitioning mode information indicates the weighting coefficient of the square partitioning mode.

[0189] The multiple hypothesis information list, i.e., combination[], includes, but is not limited to, the following examples, where one example includes multiple combinations. Each combination includes {parameters for the triangle partitioning pattern or the square partitioning pattern, index of the candidate list of fused motion information for the first hypothesis, index of the candidate list of fused motion information for the second hypothesis, and partitioning pattern flag bit}. Of course, this application does not limit the order of these four parameters in each combination; that is, in examples not listed, the order of the indicated information may differ.

[0190] In this application, when dividing into squares, the indexes of the first hypothetical fusion motion information candidate list and the second hypothetical fusion motion information candidate list use a preset combination of indices, including but not limited to the combination of index and index+1, as well as the combination of index+m and index+n, where m and n are both less than the length of the fusion motion information candidate list, and m is not equal to n. Also, since the parameter list for triangular division stores a variety of specific implementation methods, for example, about 40, a portion of the implementation methods can be randomly selected or specified as parameters for triangular division in the multi-hypothesis information list described in this application.

[0191] Each of Examples 1 through 4 includes 10 multiple hypothesis statements.

[0192] Example 1:

[0193] combination[] =

[0194] {

[0195] {0,1,0,0},{1,0,1,0},{0,1,2,0},{1,1,0,1},{1,0,2,0},

[0196] {1,1,2,1},{0,2,0,0},{0,3,0,1},{0,1,3,0},{0,2,1,0},

[0197] }

[0198] Example 2:

[0199] combination[] =

[0200] {

[0201] {0,1,0,0},{1,0,1,1},{0,1,2,0},{1,1,0,0},{1,0,2,0},

[0202] {1,1,2,1},{0,2,0,1},{0,3,0,0},{0,1,3,0},{0,2,1,0}

[0203] }

[0204] Example 3:

[0205] combination[] =

[0206] {

[0207] {0,1,0,0},{1,0,1,1},{1,0,1,0},{0,1,2,0},{0,1,2,1},

[0208] {1,1,0,0},{1,0,2,0},{1,1,2,0},{0,3,0,0},{0,3,0,1}

[0209] }

[0210] Example 4:

[0211] combination[] =

[0212] {

[0213] {0,1,0,0},{0,1,0,1},{1,0,1,0},{1,0,1,1},{0,1,2,0},

[0214] {0,1,2,1},{1,1,0,0},{1,1,0,1},{1,1,0,0},{1,1,0,1}

[0215] }

[0216] If all image blocks to be decoded are partitioned using only triangles, there are numerous partitioning methods (over 40), and triangles have higher processing complexity than squares, resulting in low efficiency in obtaining the predicted image block for the current block. Conversely, if all image blocks to be decoded use only square partitioning, the combinations of additional hypothetical motion information are limited, and since both the original and additional hypotheses use square partitioning, the method of multi-hypothesis fusion is relatively simple, leading to poor encoding and decoding performance. In this application, a combination only requires 10 specific partitioning methods (one partitioning method corresponds to one set of multi-hypothesis information) to meet the encoding and decoding requirements, reducing complexity compared to using only triangle partitioning while ensuring decoding efficiency and performance.

[0217] Using the method of this application, since the bitstream carries a partitioning identifier indicating the multi-hypothesis method, the decoding end can determine which partitioning method is used to predict the image in the multi-hypothesis method. In other words, different partitioning methods can be used to obtain the prediction block of the image block for different image blocks. In this way, the image block can use a partitioning method that is more suitable for its own characteristics, thereby improving the efficiency of image prediction and reducing the decoding time.

[0218] If the multi-hypothesis mode information indicates that the current block does not use the multi-hypothesis coding mode, then execute:

[0219] If the current block is in merge / skip mode, a fusion motion information candidate list is generated. Specifically, this includes adding spatial and temporal candidates from the current block to the fusion motion information candidate list, using the same method as in HEVC. Figure 1 As shown, spatial fusion candidates include A0, A1, B0, B1, and B2, while temporal fusion candidates include T0 and T1. In VTM, temporal fusion candidates also include candidates provided by the Adaptive Temporal Motion Vector Prediction (ATMVP) technique. This invention does not involve the process of generating a fusion motion information candidate list, which can be performed using methods in HEVC or VTM, or other methods for generating fusion motion information candidate lists. If the current block is in InterMVP mode, generating a motion vector prediction candidate list is existing technology and can be performed using methods in HEVC or VTM, or other methods for generating motion vector prediction candidate lists; this invention does not limit this.

[0220] Then, the motion information of the first hypothesis is obtained. At the decoding end: if the current block is in merge / skip mode, the motion information of the current block is determined according to the fusion index carried in the bitstream. If the current block is in Inter MVP mode, the motion information of the current block is determined according to the inter-frame prediction direction, reference frame index, motion vector prediction value index, and motion vector residual value transmitted in the bitstream.

[0221] Step 3: Based on the motion information of the first and second hypotheses, as well as the partitioning pattern identifier and partitioning pattern information, perform inter-frame prediction to obtain the predicted image of the current block.

[0222] Among them, the multi-hypothesis coding mode adopted by the current block is determined based on the multi-hypothesis mode information.

[0223] The first and second hypotheses are predicted using motion information. At the decoder: using the reference frame direction, reference frame number, and motion vector from the motion information of the first and second hypotheses, the reference blocks for the first and second hypotheses are located in the reference frame. The predicted blocks for the first and second hypotheses are then obtained based on the multi-hypothesis combination mode information. Forward prediction using the reference frame direction means the current coding unit selects a reference image from the forward reference image set to obtain a reference block. Backward prediction using the reference frame direction means the current coding unit selects a reference image from the backward reference image set to obtain a reference block. Bidirectional prediction using the reference frame direction means selecting a reference image from both the forward and backward reference image sets to obtain a reference block. When using the bidirectional prediction method, the current coding unit will have two reference blocks, each requiring its own motion vector and reference frame index for indication.

[0224] Then, the pixel values ​​of the predicted pixels in the current block are determined based on the pixel values ​​of the pixels in the reference block. More specifically, one of the following methods can be used:

[0225] Method 1: When using the triangular multi-hypothesis mode, triangular prediction blocks for the first and second hypotheses are obtained based on the directions of the multiple hypotheses. A final predicted image is generated from the first and second hypotheses using preset weighting coefficients. These weighting coefficients can be referenced in JVET-K0144, and are not limited in this invention. When using the square multi-hypothesis mode, square prediction blocks for the first and second hypotheses are obtained. A final predicted image is generated from the first and second hypotheses using preset weighting coefficients. These weighting coefficients can be referenced in JVET-K0257, and are not limited in this invention.

[0226] Method 1: When using the triangular multi-hypothesis mode, square prediction blocks for the first and second hypotheses are obtained based on the directions of the multiple hypotheses. A final predicted image is generated from the first and second hypotheses using preset weighting coefficients. The weighting coefficients can be determined using a triangular weighting coefficient matrix as described in JVET-K0144, and are not limited in this invention. When using the square multi-hypothesis mode, square prediction blocks for the first and second hypotheses are obtained. A final predicted image is generated from the first and second hypotheses using preset weighting coefficients. The weighting coefficients can be determined as described in JVET-K0257, and are not limited in this invention.

[0227] If the multi-hypothesis mode information indicates that the current block does not use the multi-hypothesis coding mode, then the motion information of the first hypothesis is used to perform inter-frame prediction to obtain the predicted image of the current block.

[0228] Decoding end: Using the reference frame direction, reference frame number, and motion vector, the prediction block is obtained from the reference frame. Forward prediction using the reference frame direction means the current coding unit selects a reference image from the forward reference image set to obtain the reference block. Backward prediction using the reference frame direction means the current coding unit selects a reference image from the backward reference image set to obtain the reference block. Bidirectional prediction using the reference frame direction means selecting a reference image from both the forward and backward reference image sets to obtain the reference block. When using the bidirectional prediction method, the current coding unit will have two reference blocks, each requiring its own motion vector and reference frame index for indication. Then, the pixel values ​​of the pixels in the prediction block of the current block are determined based on the pixel values ​​of the pixels in the reference blocks. This can be performed using methods from HEVC or VTM, or other methods for generating motion vector prediction candidate lists; this invention is not limited to these methods.

[0229] Step 4: Add the final inter-frame predicted image of the current block (that is, the predicted image block of the current image block to be decoded mentioned above) to the residual image to obtain the reconstructed image of the current block;

[0230] Decoding end: If the current block has a residual, the residual information and the predicted image are added together to obtain the reconstructed image of the current block; if the current block has no residual, the predicted image is the reconstructed image of the current block.

[0231] The above process is existing technology. For example, the same method as HEVC or VTM can be used, or other motion compensation and image reconstruction methods can be used. This invention does not limit the scope of the invention.

[0232] This application also describes an image prediction apparatus that can be used to perform the image prediction method described above.

[0233] In one implementation, the image prediction device includes: a memory for storing video data in the form of a bitstream, the video data including one or more image blocks; and a video decoder for performing the various image prediction methods described above (e.g., Figure 6In one implementation of the corresponding decoding method, a video decoder is used to parse the bitstream to obtain the multi-hypothesis information index of the current image block to be decoded; based on the multi-hypothesis information index, it obtains the first multi-hypothesis information corresponding to the current image block from the multi-hypothesis information list, the first multi-hypothesis information including a motion information index and a first identifier, the motion information index of the first multi-hypothesis information indicating the multi-hypothesis motion information of the current image block to be decoded, the first identifier indicating a first partitioning method of the multi-hypothesis method, and the first multi-hypothesis information also including parameters of the first partitioning method; motion compensation is performed based on the motion information encoded by the multi-hypothesis of the current image block to be decoded to obtain the predicted image blocks of multiple hypotheses; based on the first identifier and the parameters of the first partitioning method, the predicted image blocks of the multiple hypotheses are processed using the first partitioning method to obtain the predicted image block of the current image block to be decoded.

[0234] Of course, the video decoder can also perform... Figure 6 Other implementations of the corresponding method will not be elaborated here. Furthermore, it is clear that the video decoder can be... Figure 5 The processor in the running Figure 5 The code implementation in the memory.

[0235] In another implementation, such as Figure 7 As shown, the image prediction device 700 includes a parsing module 701, which is used to parse the bitstream to obtain the multi-hypothesis information index of the current image block to be decoded.

[0236] The first query module 702 is used to obtain the first multiple hypothesis information corresponding to the current image block from the multiple hypothesis information list according to the multiple hypothesis information index. The first multiple hypothesis information includes a motion information index and a first identifier. The motion information index of the first multiple hypothesis information indicates the multiple hypothesis motion information of the current image block to be decoded. The first identifier indicates the first partitioning method of the multiple hypothesis method. The first multiple hypothesis information also includes parameters of the first partitioning method.

[0237] The motion compensation module 703 is used to perform motion compensation based on the motion information of the multi-hypothesis encoding of the current image block to be decoded, so as to obtain the predicted image block of multiple hypotheses.

[0238] The segmentation processing module 704 is used to process the plurality of hypothetical predicted image blocks according to the parameters of the first identifier and the first segmentation method, using the first segmentation method, to obtain the predicted image block of the current image block to be decoded.

[0239] The above module division is only illustrative. In actual implementation, the functions of multiple modules can be implemented by one module.

[0240] The image prediction device 700 can perform... Figure 6 The corresponding methods and other implementations of the methods mentioned above will not be elaborated here.

[0241] Using the image prediction device described above, since the bitstream carries a partitioning identifier indicating the multiple hypothesis method, the decoder can determine which partitioning method is used to predict the image in the multiple hypothesis method. In other words, different partitioning methods can be used to obtain the predicted blocks of different image blocks. This allows the image blocks to use a partitioning method that is more suitable for their own characteristics, thereby improving the efficiency of image prediction and reducing the decoding time.

[0242] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0243] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0244] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0245] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0246] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0247] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0248] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image encoding method, characterized in that, The image encoding method includes: Obtain the prediction mode information, first list index, second list index, and parameters of the first partitioning method for the current image patch; A candidate list of fused motion information is obtained based on the predicted pattern information; The first motion information is confirmed from the fusion motion information candidate list based on the first list index; The second motion information is confirmed from the candidate list of fused motion information based on the index of the second list; The first list index and the second list index indicate the motion information of the current image block, and the parameters of the first division method indicate the division method of the current image block; Based on the parameters of the first list index, the second list index, and the first partitioning method, multiple predicted image patches are obtained; Based on the plurality of predicted image blocks, the predicted image block of the current image block is obtained; The prediction mode information, the first list index, the second list index, and the parameters of the first partitioning method are encoded into the bitstream.

2. The method according to claim 1, wherein the first list index and the second list index have different values.

3. The method according to any one of claims 1 to 2, characterized in that, The first division method is a triangular division, and the parameters of the first division method indicate the division direction of the triangle.

4. The method according to any one of claims 1 to 2, characterized in that, The first division method is a square division, and the parameters of the first division method indicate the weighting coefficients of the square division.

5. The method according to any one of claims 1 to 4, characterized in that, The first list index and the second list index indicate the motion information of the current image block, including: Based on the first list index, original motion information is obtained from the first candidate motion information list of the current image block, and additional motion information is obtained from the first candidate motion information list of the current image block based on the second list index.

6. An image decoding method, characterized in that, The image decoding method includes: Parse the bitstream to obtain the prediction mode information, first list index, second list index, and parameters of the first partitioning method for the current image patch; A candidate list of fused motion information is obtained based on the predicted pattern information; The first motion information is confirmed from the fusion motion information candidate list based on the first list index; The second motion information is confirmed from the candidate list of fused motion information based on the index of the second list; The first list index and the second list index indicate the motion information of the current image block, and the parameters of the first division method indicate the division method of the current image block; Based on the parameters of the first list index, the second list index, and the first partitioning method, multiple predicted image patches are obtained; Based on the plurality of predicted image blocks, the predicted image block of the current image block is obtained.

7. The method according to claim 6, wherein the first list index and the second list index have different values.

8. The method according to any one of claims 6 to 7, characterized in that, The first division method is a triangular division, and the parameters of the first division method indicate the division direction of the triangle.

9. The method according to any one of claims 6 to 7, characterized in that, The first division method is a square division, and the parameters of the first division method indicate the weighting coefficients of the square division.

10. The method according to any one of claims 6 to 9, characterized in that, The first list index and the second list index indicate the motion information of the current image patch, including: Based on the first list index, original motion information is obtained from the first candidate motion information list of the current image block, and additional motion information is obtained from the first candidate motion information list of the current image block based on the second list index.

11. An image decoding device, characterized in that, The image decoding device includes: The memory is used to store video data in bitstream format; The video decoder is used to parse the bitstream to obtain the prediction mode information, first list index, second list index, and parameters of the first partitioning method for the current image block; A candidate list of fused motion information is obtained based on the predicted pattern information; The first motion information is confirmed from the fusion motion information candidate list based on the first list index; The second motion information is confirmed from the candidate list of fused motion information based on the index of the second list; The first list index and the second list index indicate the motion information of the current image block, and the parameters of the first division method indicate the division method of the current image block; Based on the parameters of the first list index, the second list index, and the first partitioning method, multiple predicted image patches are obtained; Based on the plurality of predicted image blocks, the predicted image block of the current image block is obtained.

12. An image encoding device, characterized in that, The image encoding device includes: A video encoder is used to obtain the prediction mode information, first list index, second list index, and parameters of the first partitioning method for the current image patch; A candidate list of fused motion information is obtained based on the predicted pattern information; The first motion information is confirmed from the fusion motion information candidate list based on the first list index; The second motion information is confirmed from the candidate list of fused motion information based on the index of the second list; The first list index and the second list index indicate the motion information of the current image block, and the parameters of the first division method indicate the division method of the current image block; Based on the parameters of the first list index, the second list index, and the first partitioning method, multiple predicted image patches are obtained; Based on the plurality of predicted image blocks, the predicted image block of the current image block is obtained; The prediction mode information, the first list index, the second list index, and the parameters of the first partitioning method are encoded into the bitstream.

13. A terminal, characterized in that, The terminal includes: one or more processors, memory, and communication interfaces; The memory and the communication interface are connected to the one or more processors; the terminal communicates with other devices through the communication interface; the memory is used to store computer program code, the computer program code including instructions; when the one or more processors execute the instructions, the terminal executes the decoding method as described in any one of claims 6-10 or the encoding method as described in any one of claims 1-5.

14. A storage medium, characterized in that, The storage medium includes a bitstream generated by the method described in any one of claims 1-5.

15. A video decoder, comprising a non-volatile storage medium and a central processing unit, characterized in that, The non-volatile storage medium stores an executable program, and the central processing unit is connected to the non-volatile storage medium. When the central processing unit executes the executable program, the video decoder executes the decoding method as described in any one of claims 6-10.

Images