Methods for encoding and decoding images and related apparatus and systems

Abstract

The present application relates to a method for encoding an image. In some embodiments, the method comprises: (i) determining a width and a height of a coding block in the image; (ii) if the width and the height are equal to N, determining a matrix-based intra prediction (MIP) size identifier, the MIP size identifier indicating that a MIP prediction size is equal to N, wherein N is a positive integer power of 2; (iii) determining a set of reference samples of the coding block; (iv) determining a MIP prediction value of the coding block based on the set of reference samples and a MIP matrix corresponding to the MIP size identifier.

Description

Technical Field

[0001] This application relates to the field of telecommunications technology, and more particularly to methods for encoding and decoding images (e.g., pictures or videos). Background Technology

[0002] Versatile video coding (VVC) is a next-generation video compression standard intended to replace current standards such as H.265 / High Efficiency Video Coding (HEVC). VVC offers higher coding quality than current standards. To achieve this, various intra-frame and inter-frame prediction modes are considered. When using these prediction modes, video can be compressed, thus reducing the amount of data transmitted as a bitstream (binary form). Matrix-based intra-frame prediction (MIP) is one such mode. When implemented in MIP mode, the encoder (or codec) or decoder can predict intra-frame prediction blocks based on the current block (e.g., a set of bits or numbers transmitted as a unit and that can be encoded and / or decoded together). However, predicting such blocks can require significant computational resources and additional storage space. Therefore, improved methods to address this problem are advantageous and necessary. Summary of the Invention

[0003] In the current MIP mode, to generate the prediction block for the current block, the size of the prediction block is smaller than the size of the current block. For example, an "8x8" current block can have a "4x4" prediction block. In the current MIP mode, a MIP prediction block smaller than the current block size is obtained through matrix computation, which consumes fewer computational resources than performing matrix computation with a larger block. After matrix computation, the MIP prediction block is upsampled to obtain an intra-prediction block with the same size as the current block. For example, an "8x8" intra-prediction block can be determined from a "4x4" MIP prediction block by calling an upsampling process of interpolation and / or extrapolation. This application provides a method for implementing MIP mode without an upsampling process, thereby significantly reducing computational complexity and improving overall efficiency. More specifically, when implementing MIP mode, this method determines an appropriate size identifier (or MIP size identifier) ​​such that the size of the MIP prediction block (e.g., "8x8") is the same as the size of the current block ("8x8"), thus eliminating the need to perform an upsampling process.

[0004] This application provides a method for encoding an image. The method can also be applied to encoding a video containing a sequence of images. The method includes, for example, (i) determining the width and height of a coded block (e.g., a coded block) in the image; (ii) if the width and height are “N” (“N” is a positive power of 2), determining a matrix-based intra-prediction (MIP) size identifier indicating that the MIP prediction size is equal to “N”; (iii) determining a set of reference samples for the coded block (e.g., using neighboring samples of the coded block); (iv) using the set of reference samples and a MIP weight matrix based on the MIP size identifier to determine the MIP prediction value of the coded block; and (v) setting the prediction value of the coded block to be equal to the MIP prediction value of the coded block. In some embodiments, the method further includes generating a bitstream based on the prediction value of the coded block.

[0005] According to another aspect of this application, a method for decoding an image may include, for example, (a) parsing a bitstream to determine the width, height, and prediction mode of a decoding block (e.g., whether the bitstream indicates that a MIP mode is used); (b) if the width and height are “N” and a MIP mode is used, determining a MIP size identifier that indicates that the MIP prediction size is equal to “N” (“N” is a positive power of 2); (c) determining a set of reference samples for the decoding block (e.g., using neighboring samples of the decoding block); (d) determining the MIP prediction value of the decoding block using the set of reference samples and a MIP weight matrix based on the MIP size identifier; and (e) setting the prediction value of the decoding block to be equal to the MIP prediction value of the decoding block.

[0006] In some embodiments, the MIP prediction values ​​may include the values ​​of an “NxN” predicted sample (e.g., “8x8”). In some embodiments, the MIP matrix may be selected from a set of predefined MIP matrices.

[0007] Another aspect of this application includes a system for encoding / decoding images and videos. The system may include an encoding subsystem (or encoder) and a decoding subsystem (or decoder). The encoding subsystem includes a partitioning unit, a first prediction unit, and an entropy coding unit. The partitioning unit receives the input video and divides it into one or more coding units (CUs). The first intra-frame prediction unit generates prediction blocks corresponding to each CU and MIP size identifiers determined from encoding the input video. The entropy coding unit transforms parameters to write the prediction blocks into the bitstream. The decoding subsystem includes a parsing unit and a second intra-frame prediction unit. The parsing unit parses the bitstream to obtain numerical values ​​(e.g., values ​​associated with one or more CUs). The second intra-frame prediction unit converts these numerical values ​​into output video, at least partially based on the MIP size identifiers.

[0008] The width and height of a CU can be equal to "N", where "N" is a positive integer power of 2. The MIP size identifier indicates that the MIP prediction size used by the first intra-frame prediction unit to generate the MIP prediction block is "N". For example, an MIP size identifier of "2" indicates that the MIP prediction size is "8x8". Attached Figure Description

[0009] To make the technical solutions described in the embodiments of this application clearer, the accompanying drawings used to describe the embodiments are briefly explained. Obviously, the drawings described below are for illustration only and are not intended to be limiting. It should be understood that those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0010] Figure 1 This is a schematic diagram of a system according to an embodiment of this application.

[0011] Figure 2 This is a schematic diagram of an encoding system according to an embodiment of this application.

[0012] Figure 3 This is a schematic diagram illustrating the determination of intra-frame prediction blocks using the MIP mode according to an embodiment of this application.

[0013] Figure 4 This is a schematic diagram of a decoding system according to an embodiment of this application.

[0014] Figure 5 This is a schematic diagram of an apparatus (e.g., an encoder) according to an embodiment of this application.

[0015] Figure 6 This is a schematic diagram of an apparatus (e.g., a decoder) according to an embodiment of this application.

[0016] Figure 7 This is a schematic diagram of a communication system according to an embodiment of this application. Detailed Implementation

[0017] To facilitate understanding of this application, a more comprehensive description of this application will be provided below in conjunction with the accompanying drawings.

[0018] Figure 1 This is a schematic diagram of a system 100 according to an embodiment of this application. System 100 can encode, transmit, and decode images. System 100 can also be applied to encode, transmit, and decode video containing image sequences. More specifically, system 100 can receive input images, process input images, and generate output images. System 100 includes an encoding device 100a and a decoding device 100b. Encoding device 100a includes a partitioning unit 101, a first intra-frame prediction unit 103, and an entropy coding unit 105. Decoding device 100b includes a parsing unit 107 and a second intra-frame prediction unit 109.

[0019] The partitioning unit 101 receives the input video 10 and then partitions it into one or more coding tree units (CTUs) or coding units (CUs) 12. The CUs 12 are sent to the first intra-frame prediction unit 103. The first intra-frame prediction unit 103 determines the prediction block for each CU 12 by performing a MIP process. Based on the size of the CU 12, the MIP process uses different processing methods for CUs of different sizes. For each type of CU 12, a specified MIP size identifier (e.g., 0, 1, 2, etc.) is provided. The MIP size identifier is used to determine the size of the MIP prediction block (i.e., the variable "predSize"), the number of reference samples from the upper or left boundary of the CU (i.e., the variable "boundarySize"), and to select a MIP matrix from multiple predefined MIP matrices. For example, when the MIP size identifier is "0", the size of the MIP prediction block is "4x4" (e.g., "predSize" is set to equal 4) and "boundarySize" is set to equal 2; when the MIP size identifier is "1", "predSize" is set to equal 4 and "boundarySize" is set to equal 2; when the MIP size is 2, "predSize" is set to equal 8 and "boundarySize" is set to equal 4.

[0020] The first intra-frame prediction unit 103 first determines the width and height of CU 12. For example, the first intra-frame prediction unit 103 may determine that the height of CU 12 is "8" and the width is "8". In this example, the width and height are both "8". Accordingly, the first intra-frame prediction unit 103 determines the MIP size identifier of CU 12 to be "2", which indicates that the size of the MIP prediction is "8x8". The first intra-frame prediction unit 103 also determines a set of reference samples for CU 12 (e.g., determined using neighboring samples of CU 12, such as the top neighbor or left neighbor, reference). Figure 3 (Detailed discussion follows). The first intra-prediction unit 103 then determines the MIP prediction value of CU 12 based on the set of reference samples and the corresponding MIP matrix. The first intra-prediction unit 103 can use the MIP prediction value as the intra-prediction value 14 of CU 12. The intra-prediction value 14 and the parameters used to determine the intra-prediction value 14 are then sent to the entropy coding unit 105 for further processing.

[0021] Entropy coding unit 105 is used to convert the parameters used to determine the intra-frame prediction value 14 into binary form. Accordingly, entropy coding unit 105 generates bitstream 16 based on the intra-frame prediction value 14. In some embodiments, bitstream 16 can be transmitted via a communication network or stored on a disk or server.

[0022] Decoding device 100b receives bitstream 16 as input bitstream 17. Parsing unit 107 parses the input bitstream 17 (in binary form) and converts it into numerical values ​​18. Numerical value 18 indicates characteristics of the input video 10 (e.g., color, brightness, depth, etc.). Numerical value 18 is sent to second intra-frame prediction unit 109. The second intra-frame prediction unit 109 can then convert these numerical values ​​18 into output video 19 (e.g., based on processes similar to those performed by the first intra-frame prediction unit 103; see related embodiments for details). Figure 4 (See detailed discussion). The output video 19 can then be stored, transmitted, and / or presented by an external device (e.g., memory, transmitter, etc.). The stored video can also be displayed on a monitor.

[0023] Figure 2This is a schematic diagram of an encoding system 200 according to an embodiment of this application. The encoding system 200 is used to encode, compress, and / or process an input image 20, and generate an output bitstream 21 in binary form. The encoding system 200 includes a partitioning unit 201, which is used to divide the input image 20 into one or more coding tree units (CTUs) 22. In some embodiments, the partitioning unit 201 can divide the image into slices, tiles, and / or bricks. Each brick can contain one or more complete and / or partial CTUs 22. In some embodiments, the partitioning unit 201 can also form one or more sub-images, each sub-image can contain one or more slices, tiles, or bricks. The partitioning unit 201 sends the CTUs 22 to a prediction unit 202 for further processing.

[0024] Prediction unit 202 is used to generate prediction block 23 for each CTU 22. Prediction block 23 can be generated based on one or more inter-frame or intra-frame prediction methods using various interpolation and / or extrapolation schemes. For example... Figure 2 As shown, the prediction unit 202 may further include a block partitioning unit 203, a motion estimation (ME) unit 204, a motion compensation (MC) unit 205, and an intra-frame prediction unit 206. The block partitioning unit 203 is used to divide the CTU 22 into smaller coding units (CUs) or coding blocks (CBs). In some embodiments, CUs can be generated from the CTU 22 using various methods, such as quadtree partitioning, binary partitioning, and ternary partitioning. The ME unit 204 is used to estimate changes caused by the motion of the object shown in the input image 20 or the motion of the image acquisition device that generated the input image 20. The MC unit 205 is used to adjust and compensate for the changes caused by the aforementioned motion. Both the ME unit 204 and the MC unit 205 are used to determine inter-frame (e.g., at different time points) prediction blocks of the CUs. In some embodiments, the ME unit 204 and the MC unit 205 may use a rate-distortion optimized motion estimation method to determine inter-frame prediction blocks.

[0025] Intra-prediction unit 206 is used to determine inter-frame (e.g., at the same time point) prediction blocks of the CU (or a portion of the CU) using various intra-prediction modes (including MIP mode). (Refer to...) Figure 3 The details of determining intra-prediction blocks using the MIP mode (hereinafter referred to as the "MIP procedure") are discussed. In the MIP procedure, the intra-prediction unit 206 first determines one or more reference samples from the neighboring samples of the CU, for example, by directly using neighboring samples as reference samples, downsampling neighboring samples, or directly extracting from neighboring samples (e.g., ...). Figure 3 of Step 301).

[0026] Secondly, the intra-prediction unit 206 uses reference samples, the MIP matrix, and shift parameters to determine prediction samples at multiple sample positions within the CU. The sample positions can be preset sample positions within the CU. For example, a sample position can be a position within the CU with odd horizontal and vertical coordinate values ​​(e.g., x = 1, 3, 5, etc., y = 1, 3, 5, etc.). The shift parameters include shift offset parameters and shift amount parameters, which can be used for shift operations when generating prediction samples. Through this arrangement, the intra-prediction unit 206 can generate prediction samples within the CU (i.e., "MIP prediction" or "MIP prediction block" refers to a set of such prediction samples) (e.g. Figure 3 of Step 302 In some embodiments, the sample location can be a location within the CU that has an even number of horizontal and vertical coordinate values.

[0027] Third, the intra-frame prediction unit 206 can determine the predicted sample values ​​at the remaining locations of the CU (e.g., locations that are not sample locations). Figure 3 of Step 303 In some embodiments, the intra-prediction unit 206 may use an interpolation filter to determine the predicted sample values ​​at the remaining locations. Through the above processing, the intra-prediction unit 206 can generate the prediction block 23 of the CU in the CTU 22.

[0028] refer to Figure 2 Prediction unit 202 outputs prediction block 23 to adder 207. Adder 207 calculates the difference between the output of partitioning unit 201 (e.g., CU in CTU 22) and the output of prediction unit 202 (i.e., prediction block 23 of CU). Transform unit 208 reads the residual R and performs one or more transform operations on prediction block 23 to obtain coefficients 24 for further use. Quantization unit 209 can quantize coefficients 24 and output quantized coefficients 25 (e.g., levels) to dequantization unit 210. Dequantization unit 210 performs dequantization on quantized coefficients 25 to output reconstructed coefficients 26 to inverse transform unit 211. Inverse transform unit 211 performs one or more inverse transforms corresponding to the transforms in transform unit 208 and outputs reconstructed residual 27.

[0029] Then, adder 212 computes the reconstructed CU by adding the reconstructed residual 27 to the predicted block 23 of the CU from prediction unit 202. Adder 212 also forwards its output 28 to prediction unit 202 for use as an intra-frame prediction reference. After all CUs of CTU 22 have been reconstructed, filtering unit 213 can perform intra-loop filtering on the reconstructed image 29. Filtering unit 213 contains one or more filters, such as deblocking filters, sample adaptive offset (SAO) filters, adaptive loop filters (ALF), luma mapping with chroma scaling (LMCS) filters, neural network-based filters, and other suitable filters, to suppress coding distortion or enhance the coding quality of the image.

[0030] The filtering unit 213 can then send the decoded image 30 (or sub-image) to the decoded image buffer (DPB) 214. The DPB 214 outputs the decoded image 31 based on the control information. The image 31 stored in the DPB 214 can also be used as a reference image for the prediction unit 202 to perform inter-frame or intra-frame prediction.

[0031] The entropy coding unit 215 is used to convert the image 31, parameters from units in the coding system 200, and supplementary information (e.g., information for controlling or communicating with the system 200) into binary form. The entropy coding unit 215 can then generate an output bitstream 21.

[0032] In some embodiments, the encoding system 200 may be a computing device having a processor and a storage medium containing one or more encoding programs. When the processor reads and executes the encoding program, the encoding system 200 may receive an input image 20 and generate an output bitstream 21 accordingly. In some embodiments, the encoding system 200 may be a computing device having one or more chips. The units or elements of the encoding system 200 may be implemented as on-chip integrated circuits.

[0033] Figure 3 This is a schematic diagram of the MIP process according to an embodiment of this application. The MIP process can be implemented by an intra-frame prediction unit (e.g., intra-frame prediction unit 206). Figure 3 As shown, the intra-frame prediction unit may include a prediction module 301 and a filtering module 302. For example... Figure 3 As shown, the MIP process includes three steps 301, 302, and 303. The MIP process can generate a prediction block based on the current block or coding block 300 (e.g., CU or a partition of CU).

[0034] Step 301

[0035] In step 301, the intra-prediction unit can generate reference samples 32 and 34 using neighboring samples 31 and 33 of the coding block 300. In the illustrated embodiment, neighboring sample 31 is the upper neighbor sample, and neighboring sample 33 is the left neighbor sample. The intra-prediction unit 206 can calculate the average value of every two neighboring samples 31 and 33, and set these average values ​​as the values ​​of the reference samples 32 and 34, respectively. In some embodiments, the intra-prediction unit 206 can select the value of one of every two neighboring samples 31 or 33 as the value of the reference sample 32 or 34. In the illustrated embodiment, the intra-prediction unit 206 determines four reference samples 32 from the eight upper neighboring samples 31 of the coding block 300, and determines another four reference samples 34 from the eight left neighboring samples 33 of the coding block 300.

[0036] In step 301, the intra-prediction unit determines the width and height of the coded block 300, and denotes them as variables "cbWidth" and "cbHeight," respectively. In some embodiments, the intra-prediction unit 206 may employ a rate-distortion optimization mode decision process to determine the intra-prediction mode (e.g., whether to use MIP mode). In such embodiments, the coded block 300 may be divided into one or more transform blocks, the width and height of which are denoteed as variables "nTbW" and "nTbH," respectively. When MIP mode is used as the intra-prediction mode, the intra-prediction unit determines the MIP size identifier (denoted as variable "mipSizeId") based on the following condition AC.

[0037] [Condition A] If both “nTbW” and “nTbH” are 4, then set “mipSizeId” to 0.

[0038] [Condition B] Otherwise, if “cbWidth” or “cbHeight” is 4, then set “mipSizeId” to 1.

[0039] [Condition C] Otherwise, set “mipSizeId” to 2.

[0040] For example, if the size of coded block 300 is "8x8" (i.e., "cbWidth" and "cbHeight" are both 8), then "mipSizeId" is set to 2. As another example, if the size of the transform block of coded block 300 is "4x4" (i.e., "nTbW" and "nTbH" are both 4), then "mipSizeId" is set to 0. And as yet another example, if the size of coded block 300 is "4x8", then "mipSizeId" is set to 1.

[0041] In the illustrated embodiment, there are three "mipSizeId" values: "0", "1", and "2". Each MIP size identifier (i.e., the variable "mipSizeId") corresponds to a specific way of performing the MIP process (e.g., using a different MIP matrix). In other embodiments, there may be more than three MIP size identifiers.

[0042] Based on the MIP size identifier, the intra-prediction unit can determine the variables “boundarySize” and “predSize” based on the following conditional DF.

[0043] [Condition D] If “mipSizeId” is 0, set “boundarySize” to 2 and “predSize” to 4.

[0044] [Condition E] If “mipSizeId” is 1, set “boundarySize” to 4 and “predSize” to 4.

[0045] [Condition F] If “mipSizeId” is 2, set “boundarySize” to 4 and “predSize” to 8.

[0046] In the illustrated embodiment, "boundarySize" represents the number of each defined reference sample 32, 34 from the upper reference sample 31 and left reference sample 33 of the coded block 300. The variable "predSize" is used in subsequent calculations (i.e., the following equation). (C) ).

[0047] In some embodiments, the intra-prediction unit may also determine the variable "isTransposed" to indicate the order in which reference samples 32, 34 are stored in a temporary array. For example, "isTransposed" being 0 indicates that the intra-prediction unit presents reference sample 32, determined from the upper neighbor sample 31 of the coding block 300, before reference sample 34, determined from the left neighbor sample 33. Alternatively, "isTransposed" being 1 indicates that the intra-prediction unit presents reference sample 34, determined from the left neighbor sample 33 of the coding block 300, before reference sample 32, determined from the upper neighbor sample 31. In embodiments of the coding system 200, the value of "isTransposed" is sent as one of the parameters of the MIP process that is encoded and written into the bitstream (e.g., output bitstream 21). Accordingly, in this application... Figure 4 In the implementation of the decoding system 400, the value of "isTransposed" can be received from the parsing unit (e.g., parsing unit 401) by parsing the input bitstream (which may be the output bitstream 21).

[0048] The intra-frame prediction unit can also determine the variable "inSize" to indicate the number of reference samples 32 and 34 used to determine the MIP prediction value. The value of "inSize" is determined by the following equation (A). In this application, the meaning and operation of all operators in the equation are the same as the corresponding operators defined in the ITU-T H.265 standard.

[0049] inSize=(2*boundarySize)-(mipSizeId==2)? 1:0; (A)

[0050] For example, "==" is the relational operator "equal to". For example, if "mipSizeId" is 2, then "inSize" is 7 (calculated by (2*4)-1). If "mipSizeId" is 1, then "inSize" is 8 (calculated by (2*4)-0).

[0051] The intra-prediction unit can invoke the following procedure to determine a set of reference samples 32, 34, which are stored in the array p[x] ("x" ranges from "0" to "inSize-1"). The intra-prediction unit can determine "nTbW" samples from the upper neighbor sample 31 of the coded block 300 (and store them in the array "refT"), and determine "nTbH" samples from the left neighbor sample 33 of the coded block 300 (and store them in the array "refL").

[0052] The intra-prediction unit can downsample "refT" to obtain "boundarySize" samples and store these "boundarySize" samples in "refT". Similarly, the intra-prediction unit 206 can downsample "refL" to obtain "boundarySize" samples and store these "boundarySize" samples in "refL".

[0053] In some embodiments, the intra-prediction unit may merge arrays "refT" and "refL" into a single array "pTemp" based on the order indicated by the variable "isTransposed". The intra-prediction unit may determine "isTransposed" to indicate the order in which reference samples are stored in the temporary array "pTemp". For example, "isTransposed" being 0 (or FALSE) indicates that the intra-prediction unit presents reference sample 32 determined from the upper neighbor sample 31 of coding block 300 before reference sample 34 determined from the left neighbor sample 33. In other cases, "isTransposed" being 1 (or TRUE) indicates that the intra-prediction unit presents reference sample 34 determined from the left neighbor sample 33 of coding block 300 before reference sample 32 determined from the upper neighbor sample 31. In some embodiments, in implementations of the coding system 200, the intra-prediction unit may determine the value of "isTransposed" using a rate-distortion optimization method. In some embodiments of the coding system 200, the intra-frame prediction unit may determine the value of "isTransposed" based on comparisons and / or relationships between neighboring samples 32, 34 and coding block 300. In embodiments of the coding system 200, the value of "isTransposed" may be forwarded to the entropy coding unit (e.g., entropy coding unit 215) as one of the parameters of the MIP process to be written to the bitstream (e.g., output bitstream 21). Accordingly, in this application... Figure 4 In the implementation of the decoding system 400, the value of "isTransposed" can be received from the parsing unit (e.g., parsing unit 401) by parsing the input bitstream (which may be the output bitstream 21).

[0054] The intra-frame prediction unit can determine the array p[x] ("x" from "0" to "inSize-1") based on the following conditions G and H.

[0055] [Condition G] If “mipSizeId” is 2, then p[x] = pTemp[x+1] - pTemp[0].

[0056] [Condition H] Otherwise (e.g., “mipSizeId” is less than 2), then p[0] = pTemp[0] - (1 << (BitDepth-1)) and p[x] = pTemp[x] - pTemp[0] (x from 1 to “inSize-1”).

[0057] In condition H above, “BitDepth” is the bit depth of the color component (e.g., the Y component) of the sample in coded block 300. The symbol “<<” is the shift symbol used in the ITU-T H.265 standard.

[0058] Alternatively, the intra-frame prediction unit can determine the array p[x] (x ranges from 0 to "inSize-1") based on the following conditions I and J.

[0059] [Condition I] If “mipSizeId” is 2, then p[x] = pTemp[x+1] - pTemp[0].

[0060] [Condition J] Otherwise (e.g., “mipSizeId” is less than 2), then p[0] = (1 << (BitDepth-1)) - pTemp[0] and p[x] = pTemp[x] - pTemp[0] (x from 1 to “inSize-1”).

[0061] In some embodiments, the intra-prediction unit can determine the value of array p[x] using a unified calculation method without having to determine the value of “mipSizeId”. For example, the intra-prediction unit can append “(1<<(BitDepth-1))” as an additional element to “pTemp” and calculate p[x] as “pTemp[x]-pTemp[0]”.

[0062] Step 302

[0063] In step 302, the intra-frame prediction unit (or prediction module 301) determines the MIP prediction value of the coding block 300 by using the aforementioned set of reference samples 32 and 34 and the MIP matrix. The MIP matrix is ​​selected from a set of predefined MIP matrices based on the corresponding MIP mode identifier (i.e., the variable "mipModeId") and MIP size identifier (i.e., the variable "mipSizeId").

[0064] The MIP prediction value determined by the intra-frame prediction unit includes the value of part of the predicted sample 35 at all or part of the sample positions in the coded block 300. The MIP prediction value is denoted as "predMip[x][y]".

[0065] exist Figure 3 In the illustrated embodiment, the partial prediction sample 35 is the sample marked as a gray square in the current block 300. Reference samples 32 and 34 in the array p[x] determined in step 301 are used as input to the prediction module 301. The prediction module 301 calculates the partial prediction sample 35 using a MIP matrix and shift parameters. The shift parameters include a shift offset parameter and a shift amount parameter. In some embodiments, the prediction module 301 determines the partial prediction sample 35 based on the following equations (B) and (C), with coordinates (x, y):

[0066]

[0067]

[0068] In equation (B) above, the parameter “fO” is the shift offset parameter used to determine the parameter “oW”. The parameter “sW” is the shift amount parameter. “p[i]” is the reference sample. The symbol ">>” is the binary right shift operator as defined in the H.265 standard.

[0069] In the above equation (C) In this context, “mWeight[i][j]” is the MIP weighting matrix, where the matrix elements are constants that are fixed for both encoding and decoding. Alternatively, in some embodiments, the encoding system 200 is implemented using an adaptive MIP matrix. For example, the MIP weighting matrix can be updated using one or more encoded images as input, or using images provided to the encoding system 200 by an external device, through various training methods. The intra-frame prediction unit can forward “mWeight[i][j]” to the entropy coding unit (e.g., entropy coding unit 215) when the MIP mode is determined. The entropy coding unit can then write “mWeight[i][j]” into the bitstream, for example, into one or more special data units in the bitstream containing MIP data. Accordingly, in some embodiments, the decoding system 400 using the adaptive MIP matrix can update the MIP matrix using, for example, a training method, where the input to the training method is one or more encoded images or blocks, or images from other bitstreams provided to the encoding system 200 by an external device, or images obtained by parsing special data units in the input bitstream containing MIP matrix data by the parsing unit 401.

[0070] The prediction module 301 can determine the values ​​of "sW" and "fO" based on the size of the current block 300 and the MIP mode used for the current block 300. In some embodiments, the prediction module 301 can obtain the values ​​of "sW" and "fO" by using a lookup table. For example, Table 1 below can be used to determine "sW".

[0071] Table 1

[0072]

[0073] Alternatively, Table 2 below can also be used to determine "sW".

[0074] Table 2

[0075] MipSizeId sW 0 5 1 6 2 5

[0076] In some embodiments, the prediction module may set "sW" to a constant. For example, for blocks of different sizes using different MIP modes, the prediction module may set "sW" to "5". As another example, for blocks of different sizes using different MIP modes, the prediction module 301 may set "sW" to "6". Yet another example, for blocks of different sizes using different MIP modes, the prediction module may set "sW" to "7".

[0077] In some embodiments, the prediction module 301 may use Table 3 or Table 4 below to determine “fO”.

[0078] Table 3

[0079]

[0080] Table 4

[0081] MipSizeId fO 0 34 1 23 2 46

[0082] In some embodiments, the prediction module 301 can directly set "fO" to a constant (e.g., a value from 0 to 100). For example, for blocks of different sizes using different MIP modes, the prediction module 301 can set "fO" to "46". Another example is that the prediction module 301 can set "fO" to "56". Yet another example is that the prediction module 301 can set "fO" to "66".

[0083] In some embodiments, the intra-prediction unit may perform a clipping operation on the values ​​of the MIP prediction samples stored in the array "predMip". When "isTransposed" is 1 (or TRUE), the array "predMip[x][y]" (x from 0 to "predSize-1", y from 0 to "predSize-1") of "predSize x preSize" is transposed to "predTemp[y][x] = predMip[x][y]", then "predMip = predTemp".

[0084] More specifically, when the size of the coded block 303 is “8x8” (i.e., both “cbWidth” and “cbHeight” are 8), the intra-prediction unit can determine the “predMip” array of “8x8”.

[0085] Step 303

[0086] exist Figure 3 In step 303, the intra-frame prediction unit determines the predicted samples 37 for the remaining samples in the coding block 300, excluding some samples 35. For example... Figure 3As shown, the intra-frame prediction unit can use the filtering module 302 to determine the predicted samples 37 of the remaining samples in the coding block 300, excluding the partial samples 35. The input to the filtering module 302 can be the partial samples 35 from step 302. The filtering module 302 can use one or more interpolation filters to determine the predicted samples 37 of the remaining samples in the coding block 300, excluding the partial samples 35. The intra-frame prediction unit (or filtering module 302) can generate a prediction block of the coding block 300 (which includes multiple predicted samples 37) according to the following conditions K and L and store the values ​​of the predicted samples 37 in the array "predSamples[x][y]" (x from 0 to "nTbW-1", y from 0 to "nTbH-1").

[0087] [Condition K] If the intra-prediction unit determines that “nTbW” is greater than “predSize” or “nTbH” is greater than “predSize”, then the intra-prediction unit initiates an upsampling process to determine “predSamples” based on “predMip”.

[0088] [Condition L] Otherwise, the intra-prediction unit sets the prediction value of coded block 300 to the MIP prediction value of the coded block.

[0089] In other words, the intra-prediction unit can set "predSamples[x][y]" (x from 0 to "nTbW-1", y from 0 to "nTbH-1") to "predMip[x][y]". For example, the intra-prediction unit can set the "predSamples" of a coded block with a size of "8x8" (i.e., both "cbWidth" and "cbHeight" are 8) to its "predMip[x][y]".

[0090] Through steps 301-303, the intra-frame prediction unit can generate the prediction value for the current block 300. The generated prediction value can be used for further processing (e.g., as described above). Figure 2 The prediction block 23 is discussed.

[0091] Figure 4 This is a schematic diagram of a decoding system 400 according to an embodiment of this application. The decoding system 400 is used to receive and process an input bitstream 40 and transform the input bitstream 40 into an output video 41. The input bitstream 40 may be a bitstream representing a compressed / encoded image / video. In some embodiments, the input bitstream 40 may come from an output bitstream (e.g., output bitstream 21) generated by an encoding system (e.g., encoding system 200).

[0092] The decoding system 400 includes a parsing unit 401, which parses the input bitstream 40 to obtain the values ​​of the syntax elements. The parsing unit 401 also converts the binary representation of the syntax elements into numerical values ​​(i.e., decoding blocks 42) and forwards these values ​​to the prediction unit 402 (e.g., for decoding). In some embodiments, the parsing unit 401 may also forward one or more variables and / or parameters used for decoding the aforementioned numerical values ​​to the prediction unit 402.

[0093] Prediction unit 402 is used to determine prediction block 43 of decoded block 42 (e.g., CU or a portion of CU, such as a transform block). When instructed to decode the decoded block 42 using an inter-frame decoding mode, the motion compensation (MC) unit 403 of prediction unit 402 can receive relevant parameters from parsing unit 401 and perform decoding accordingly in the inter-frame decoding mode. When instructed to decode the decoded block 42 using an intra-frame prediction mode (e.g., MIP mode), the intra-frame prediction unit 404 of prediction unit 402 receives relevant parameters from parsing unit 401 and performs decoding accordingly in the indicated intra-frame decoding mode. In some embodiments, the intra-frame prediction mode (e.g., MIP mode) can be identified by a specific identifier (e.g., a MIP identifier) ​​embedded in the input bitstream 40.

[0094] For example, when MIP mode is identified, the intra-prediction unit 404 can be based on the following method (similar to...). Figure 3 The aforementioned Steps 301-303 ) Determine prediction block 43 (which includes multiple prediction samples).

[0095] First, the intra-prediction unit 404 determines one or more reference samples from the neighboring samples of the decoded block 42 (similar to...). Figure 3 In Step 301 For example, the intra-frame prediction unit 404 can generate a reference sample by downsampling neighboring samples or by directly extracting a portion of neighboring samples.

[0096] Then, the intra-prediction unit 404 can use the reference sample, the MIP matrix, and the shift parameters to determine a portion of the predicted samples in the decoding block 42 (similar to...). Figure 3 In Step 302 In some embodiments, the positions of some predicted samples can be pre-set in the decoding block 42. For example, the positions of some predicted samples can be positions within the decoding block with odd horizontal and vertical coordinate values. The shift parameters can include shift offset parameters and shift amount parameters, which can be used for shift operations when generating partial predicted samples.

[0097] Finally, if some predicted samples of decoded block 42 are determined, the intra-prediction unit 404 determines the predicted samples of the remaining samples in decoded block 42 excluding the partially predicted samples (similar to...). Figure 3 of Step 303 For example, the intra-frame prediction unit 404 can use an interpolation filter to determine the prediction samples, wherein partial prediction samples and neighboring samples are used as inputs to the interpolation filter.

[0098] The decoding system 400 includes a scaling unit 405, which functions similarly to the dequantization unit 210 of the encoding system 200. The scaling unit 405 performs a scaling operation on the quantization coefficients 44 (e.g., levels) from the parsing unit 401 to generate reconstructed coefficients 45.

[0099] The transformation unit 406 functions similarly to the inverse transformation unit 211 in the encoding system 200. The transformation unit 406 performs one or more transformation operations (e.g., the inverse operation of one or more transformation operations performed by the inverse transformation unit 211) to obtain the reconstructed residual 46.

[0100] The adder adds the predicted block 43 from the prediction unit 402 and the reconstructed residual 46 from the transform unit 406 to obtain the reconstructed block 47 of the decoded block 42. The reconstructed block 47 is also sent to the prediction unit 402 as a reference (e.g., a reference for other blocks encoded in intra-prediction mode).

[0101] After all decoded blocks 42 in the image or sub-image have been reconstructed (i.e., reconstructed blocks 48 are formed), the filtering unit 408 can perform in-loop filtering on the reconstructed blocks 48. The filtering unit includes one or more filters, such as a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a luminance mapping and chrominance scaling (LMCS) filter, a neural network-based filter, etc. In some embodiments, the filtering unit 408 can perform in-loop filtering only on one or more target pixels in the reconstructed blocks 48.

[0102] The filtering unit 408 then sends the decoded image 49 (or image) or sub-image to the decoded image buffer (DPB) 409. The DPB 409 outputs the decoded image as output video 41 based on timing and control information. The decoded image 49 stored in the DPB 409 can be used as a reference image when the prediction unit 402 performs inter-frame prediction or intra-frame prediction.

[0103] In some embodiments, the decoding system 400 may be a computing device having a processor and a storage medium storing one or more decoding programs. When the processor reads and executes the decoding program, the decoding system 400 may receive the input video stream and generate the corresponding decoded video.

[0104] In some embodiments, the decoding system 400 may be a computing device having one or more chips. Units or elements of the decoding system 400 may be implemented as on-chip integrated circuits.

[0105] Figure 5 This is a schematic diagram of an apparatus 500 according to an embodiment of this application. Apparatus 500 may be a "transmitting" apparatus. More specifically, apparatus 500 is used to acquire, encode, and store / transmit one or more images. Apparatus 500 includes an acquisition unit 501, an encoder 502, and a storage / transmission unit 503.

[0106] The acquisition unit 501 is used to acquire or receive images and forward the images to the encoder 502. The acquisition unit 501 can also be used to acquire or receive video consisting of a sequence of images and forward the video to the encoder 502. In some embodiments, the acquisition unit 501 may be a device including one or more cameras (e.g., an image camera, a depth camera, etc.). In some embodiments, the acquisition unit 501 may be a device capable of partially or completely decoding a video stream to generate images or video. The acquisition unit 501 may also include one or more elements for acquiring audio signals.

[0107] Encoder 502 is used to encode the image from acquisition unit 501 and generate a video stream. Encoder 502 can also be used to encode video from acquisition unit 501 to generate a video stream. In some embodiments, encoder 502 can be implemented as... Figure 2 The encoding system 200 is described above. In some embodiments, the encoder 502 may include one or more audio encoders to encode audio signals to generate an audio bitstream.

[0108] The storage / transmission unit 503 is used to receive video and / or audio streams from the encoder 502. The storage / transmission unit 503 can encapsulate the video and audio streams together to form a media file (e.g., an ISO-based media file) or a transport stream. In some embodiments, the storage / transmission unit 503 can write or store the media file or transport stream to a storage unit, such as a hard drive, disk, DVD, cloud storage, portable storage device, etc. In some embodiments, the storage / transmission unit 503 can transmit the video / audio stream to an external device via a transmission network (e.g., the Internet, wired network, cellular network, wireless LAN, etc.).

[0109] Figure 6 This is a schematic diagram of an apparatus 600 according to an embodiment of this application. Apparatus 600 may be a "target" apparatus. More specifically, apparatus 600 is used to receive, decode, and present images or videos. Apparatus 600 includes a receiving unit 601, a decoder 602, and a presentation unit 603.

[0110] The receiving unit 601 is used to receive media files or transport streams from, for example, a network or storage device. The media files or transport streams include video streams and / or audio streams. The receiving unit 601 can separate the video streams and audio streams. In some embodiments, the receiving unit 601 can generate new video / audio streams by extracting the video / audio streams.

[0111] Decoder 602 includes one or more video decoders, such as the decoding system 400 discussed above. Decoder 602 may also include one or more audio decoders. Decoder 602 decodes the video bitstream and / or audio bitstream from receiving unit 601 to obtain a decoded video file and / or one or more decoded audio files (corresponding to one or more channels).

[0112] The presentation unit 603 receives the decoded video / audio file and processes it to obtain a suitable video / audio signal for display / playback. These adjustment / reconstruction operations may include one or more of the following: noise reduction, compositing, color space conversion, upsampling, downsampling, etc. The presentation unit 603 can improve the quality of the decoded video / audio file.

[0113] Figure 7 This is a schematic diagram of a communication system 700 according to an embodiment of this application. The communication system 700 includes a source device 701, a storage medium or transmission network 702, and a target device 703. In some embodiments, the source device 701 may be the one referenced above. Figure 5 The described apparatus 500. Source device 701 sends media files to storage media or transmission network 702 for storage or transmission. Target device 703 may be as described above. Figure 6 The described apparatus 600. The communication system 700 is used to encode media files, transmit or store encoded media files, and then decode the encoded media files. In some embodiments, the source device 701 may be a first smartphone, the storage medium 702 may be cloud storage, and the target device may be a second smartphone.

[0114] The above embodiments are merely illustrative of several embodiments of the present invention, and their descriptions are specific and detailed. The above embodiments should not be construed as limiting this application. It should be noted that various changes and modifications can be made by those skilled in the art without departing from the spirit and scope of the present invention. Therefore, the scope of this application should be determined by the appended claims.

Claims

1. A method for encoding an image, comprising: Determine the width and height of the coded block; If the width and the height are equal to N, then a matrix-based intra-frame prediction (MIP) size identifier is determined, which indicates that the MIP prediction size is equal to N, where N is a positive integer power of 2; Determine a set of reference samples for the coded block; and Based on the set of reference samples and the MIP matrix according to the MIP size identifier, the MIP prediction value of the coded block is determined, including: The MIP prediction value of the coded block is determined based on a set of reference samples, shift offset parameters, and the MIP matrix according to the MIP size identifier. Different MIP size identifiers correspond to the same shift offset parameter, wherein the shift offset parameter is a constant value; The method further includes: The set of reference samples for the coded block is determined based on adjacent samples; Set the predicted value of the coded block to be equal to the MIP predicted value of the coded block.

2. The method according to claim 1, wherein, The step of determining the MIP prediction value of the coded block based on the set of reference samples and the MIP matrix according to the MIP size identifier further includes: The MIP prediction value of the coded block is determined based on the following equation: oW=(1<<(sW 1)) fO ( ), predMip[x][y]=((( )+oW)>>sW)+pTemp[0], x is from 0 to "predSize 1", y is from 0 to "predSize" 1” Wherein, "sW" is the shift quantity parameter, "fO" is the shift offset parameter, "inSize" indicates the number of reference samples used to determine the MIP prediction value, "p[i]" represents the reference sample, "predMip[x][y]" represents the MIP prediction value, "mWeight[i][j]" represents the MIP weighting matrix, "predSize" represents the size of the MIP prediction block, "pTemp[0]" represents the 0th value in the reference sampling buffer, the symbol "<<" represents the binary left shift operator, and the symbol ">>" represents the binary right shift operator.

3. The method according to claim 2, further comprising: The set of reference samples of the coded block is downsampled to obtain a reference sampling buffer, which contains the set of reference samples of the coded block after downsampling. The input sample value is determined based on the reference sample in the reference sampling buffer, the MIP size identifier, and the luminance bit depth.

4. The method according to claim 3, wherein, The step of determining the input sample value based on the reference sample in the reference sampling buffer, the MIP size identifier, and the luminance bit depth includes: The input sample value is derived using the following formula: If the MIP size identifier is equal to 2, then p[x] = pTemp[x+1] pTemp[0]; If the MIP size identifier is less than 2, then Where "p[x]" represents the reference sample, "pTemp[x]" represents the x-th value in the reference sampling buffer, and BitDepth is the luminance bit depth.

5. The method according to claim 4, wherein, When the width and height of the encoded block are both equal to 4, the MIP size identifier is equal to 0; When the width multiplied by the height of the encoded block is Nx4 or 4xN, the MIP size identifier is equal to 1; If the width and height of the encoded block are not equal to 4 and the product of the width and height of the encoded block is not Nx4 or 4xN, the MIP size identifier is equal to 2.

6. The method according to claim 2, wherein, The step of determining the MIP prediction value of the coded block based on the set of reference samples and the MIP matrix according to the MIP size identifier further includes: Determine the shift quantity parameter; and The MIP prediction value of the coded block is determined based on the set of reference samples, the shift quantity parameter, and the MIP matrix according to the MIP size identifier.

7. The method according to claim 6, wherein, Different MIP size identifiers correspond to the same shift number parameter.

8. The method according to claim 6, wherein, The shift quantity parameter is a constant value.

9. The method according to any one of claims 6-8, wherein, The shift quantity parameter is equal to 6.

10. The method according to claim 1, wherein, The coded block includes a coded block of size "8x8".

11. The method according to claim 1, wherein, The adjacent samples include the upper adjacent samples.

12. The method according to claim 1, wherein, The adjacent samples include the left adjacent samples.

13. The method according to claim 1, wherein, The MIP prediction values ​​include the values ​​of NxN prediction samples.

14. A method for decoding an image, comprising: Determine the width, height, and prediction mode of the decoded block; When the prediction mode indicates that the decoded block is decoded using a matrix-based intra-frame prediction (MIP) mode, if the width and the height are equal to N, a MIP size identifier is determined, which indicates that the MIP prediction size is equal to N, where N is a positive integer power of 2. Determine a set of reference samples for the decoded block; and Based on the set of reference samples and the MIP matrix corresponding to the MIP size identifier, the MIP prediction value of the decoded block is determined, including: Based on a set of reference samples, shift offset parameters, and the MIP matrix corresponding to the MIP size identifier, the MIP prediction value of the decoded block is determined; Different MIP size identifiers correspond to the same shift offset parameter, wherein the shift offset parameter is a constant value; The method further includes: The set of reference samples for the decoding block is determined based on adjacent samples; The prediction value of the decoded block is set to be equal to the MIP prediction value of the decoded block.

15. The method according to claim 14, wherein, The step of determining the MIP prediction value of the decoded block based on the set of reference samples and the MIP matrix corresponding to the MIP size identifier further includes: The MIP prediction value of the decoded block is determined based on the following equation: oW=(1<<(sW 1)) fO ( ), predMip[x][y]=((( )+oW)>>sW)+pTemp[0], x is from 0 to "predSize 1", y is from 0 to "predSize" 1” Wherein, "sW" is the shift quantity parameter, "fO" is the shift offset parameter, "inSize" indicates the number of reference samples used to determine the MIP prediction value, "p[i]" represents the reference sample, "predMip[x][y]" represents the MIP prediction value, "mWeight[i][j]" represents the MIP weighting matrix, "predSize" represents the size of the MIP prediction block, "pTemp[0]" represents the 0th value in the reference sampling buffer, the symbol "<<" represents the binary left shift operator, and the symbol ">>" represents the binary right shift operator.

16. The method of claim 15, further comprising: The set of reference samples of the decoded block is downsampled to obtain a reference sampling buffer, which contains the set of reference samples of the decoded block after downsampling. The input sample value is determined based on the reference sample in the reference sampling buffer, the MIP size identifier, and the luminance bit depth.

17. The method according to claim 16, wherein, The step of determining the input sample value based on the reference sample in the reference sampling buffer, the MIP size identifier, and the luminance bit depth includes: The input sample value is derived using the following formula: If the MIP size identifier is equal to 2, then p[x] = pTemp[x+1] pTemp[0]; If the MIP size identifier is less than 2, then Where "p[x]" represents the reference sample, "pTemp[x]" represents the x-th value in the reference sampling buffer, and BitDepth is the luminance bit depth.

18. The method according to claim 17, wherein, When the width and height of the decoded block are both equal to 4, the MIP size identifier is equal to 0; When the width multiplied by the height of the decoded block is Nx4 or 4xN, the MIP size identifier is equal to 1; The MIP size identifier is equal to 2 if the width and height of the decoded block are not equal to 4 and the product of the width and height of the decoded block is not Nx4 or 4xN.

19. The method of claim 15, wherein, The step of determining the MIP prediction value of the decoded block based on the set of reference samples and the MIP matrix corresponding to the MIP size identifier further includes: Determine the shift quantity parameter; and Based on the set of reference samples, the shift quantity parameter, and the MIP matrix corresponding to the MIP size identifier, the MIP prediction value of the decoded block is determined.

20. The method according to claim 19, wherein, Different MIP size identifiers correspond to the same shift number parameter.

21. The method according to claim 19, wherein, The shift quantity parameter is a constant value.

22. The method according to any one of claims 19-21, wherein, The shift quantity parameter is equal to 6.

23. The method according to claim 14, wherein, The decoding block includes a decoding block with a size of "8x8".

24. The method according to claim 14, wherein, The adjacent samples include the upper adjacent samples.

25. The method according to claim 14, wherein, The adjacent samples include the left adjacent samples.

26. The method according to claim 14, wherein, The MIP prediction values ​​include the values ​​of NxN prediction samples.

27. An encoding apparatus for encoding an image, comprising: A segmentation unit is used to receive an input image and divide the input image into one or more coded blocks; A prediction unit is configured to determine the width and height of the coded block, wherein if the width and height are equal to N, the prediction unit is configured to determine a matrix-based intra-frame prediction (MIP) size identifier, the MIP size identifier indicating that the MIP prediction size is equal to N, where N is a positive integer power of 2, and wherein the prediction unit is configured to determine a set of reference samples for the coded block, and the prediction unit is configured to determine the MIP prediction value of the coded block based on the set of reference samples and the MIP matrix according to the MIP size identifier; and An entropy coding unit is used to transform the parameters to write the MIP prediction value into the bitstream; Specifically, the prediction unit is used to determine the MIP prediction value of the coding block based on a set of reference samples, shift offset parameters, and the MIP matrix according to the MIP size identifier. Different MIP size identifiers correspond to the same shift offset parameter, wherein the shift offset parameter is a constant value; The prediction unit is further configured to determine the set of reference samples for the coding block based on neighboring samples; The prediction unit is further configured to set the predicted value of the coded block to be equal to the MIP predicted value of the coded block.

28. A decoding apparatus for decoding an image, comprising: The parsing unit is used to determine the width, height, and prediction mode of the decoded block; An intra-frame prediction unit is configured to, when the prediction mode indicates the use of matrix-based intra-frame prediction (MIP) mode to decode the decoded block, determine a MIP size identifier if the width and the height are equal to N, wherein the MIP size identifier indicates that the MIP prediction size is equal to N, where N is a positive integer power of 2, and wherein the intra-frame prediction unit is configured to determine a set of reference samples for the decoded block, and wherein the intra-frame prediction unit is configured to determine the MIP prediction value of the decoded block based on the set of reference samples and the MIP matrix corresponding to the MIP size identifier; Specifically, the intra-frame prediction unit is used to determine the MIP prediction value of the decoded block based on a set of reference samples, shift offset parameters, and the MIP matrix corresponding to the MIP size identifier. Different MIP size identifiers correspond to the same shift offset parameter, wherein the shift offset parameter is a constant value; The prediction unit is further configured to determine the set of reference samples for the decoded block based on neighboring samples; The prediction unit is further configured to set the prediction value of the decoded block to be equal to the MIP prediction value of the decoded block.

29. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program and a bitstream, the computer program being executed by a processor to implement the steps of the method for encoding an image according to any one of claims 1-13 to generate the bitstream.

Images