Image encoding method, decoding method, device and storage medium

Abstract

The present application provides an image encoding method, a decoding method, an apparatus and a storage medium. The image decoding method includes obtaining a code stream, and obtaining a residual image of a grid image in an image sequence according to the code stream, wherein the image sequence includes a plurality of grid images, and the geographical locations corresponding to the plurality of grid images are similar or the image contents of the plurality of grid images are similar. Inter-frame prediction is performed on the grid images in the image sequence to obtain a predicted image of the grid images in the image sequence; the grid images in the image sequence are obtained according to the residual image of the grid images in the image sequence and the predicted image of the grid images in the image sequence. The embodiments of the present application can reduce the storage overhead of the code stream.

Description

Technical Field

[0001] The present application relates to the technical field of image coding and decoding, and more specifically, to an image coding method, decoding method, device and storage medium. Background Art

[0002] Light laser detection and ranging (LiDAR) is a radar system that uses lasers as radiation sources, and can be referred to as laser radar. Laser radar is a combination of laser technology and radar technology. Laser radar has the characteristics of high detection accuracy, large measurement range, non-contact, 3D and 360-degree measurement. Laser radar is often used for high-precision map construction, high-precision positioning, target detection, 3D scene rendering, etc.

[0003] The amount of 3D point cloud data generated by LiDAR measurement is huge, which brings great challenges to the storage and transmission of point cloud data. In order to reduce the storage volume of point cloud data, 3D point cloud data is generally converted into 2D point cloud data as a positioning layer. However, the bitmap layer still covers a large geographical area, the amount of data is still large, and the code stream obtained by encoding the image in the point cloud data still occupies a large storage overhead. Summary of the invention

[0004] The present application provides an image encoding method, a decoding method, an apparatus and a storage medium to reduce the storage space occupied by a code stream.

[0005] In a first aspect, an image decoding method is provided, the method comprising: obtaining a code stream; obtaining a residual image of a grid image in an image sequence based on the code stream; performing inter-frame prediction on the grid image in the image sequence to obtain a predicted image of the grid image in the image sequence; obtaining the grid image in the image sequence based on the residual image of the grid image in the image sequence and the predicted image of the grid image in the image sequence.

[0006] The image sequence includes a plurality of grid images, and the geographical locations corresponding to the plurality of grid images are close or the image contents of the plurality of grid images are similar. Specifically, the geographical locations corresponding to the plurality of grid images are close or the image contents of the plurality of grid images are similar, which means that the plurality of grid images meet the first condition.

[0007] It should be understood that the above-mentioned inter-frame prediction of the grid image in the image sequence can be to first predict the grid image to obtain a predicted image of the grid image, and then subtract the grid image from the predicted image of the grid image to obtain a residual image of the grid image.

[0008] In addition, when performing inter-frame prediction on a grid image in an image sequence, the grid image in the image sequence may be first divided into image blocks, and then the prediction block of each image block is determined by inter-frame prediction, and then the residual block of each image block is determined. In the present application, the prediction image of the above-mentioned grid image may be composed of the prediction blocks of each image block in the grid image, and the residual image of the grid image may be composed of the residual blocks of each image block in the grid image.

[0009] In the present application, since the image sequence contains grid images with similar geographical locations or image contents, when inter-frame prediction is performed on the image sequence, the amount of data of the obtained residual image can be smaller, thereby making the storage space occupied by the code stream corresponding to the final image sequence smaller, which is convenient for storage or transmission.

[0010] In combination with the first aspect, in certain implementations of the first aspect, the first condition is that a distance between geographical locations corresponding to at least two grid images among the multiple grid images is less than a preset distance.

[0011] The above-mentioned preset distance may also be a distance threshold, and the size of the preset distance may be flexibly set according to actual needs. For example, the above-mentioned preset distance may be set to 3m (3 meters), and when the distance between the corresponding geographical locations of two grid images is less than 3m, the two grid images meet the first condition. The 3m here is only an example, and the preset distance may also be set to 5m and 6m, etc. In short, any suitable size of preset distance may be set according to needs.

[0012] In addition, the first condition may also be that a distance between geographical locations corresponding to at least two grid images among the plurality of grid images is less than or equal to a preset distance.

[0013] Since images with similar geographical locations are generally highly similar, when an image sequence includes at least two grid images with corresponding geographical locations being adjacent, performing inter-frame prediction on the image sequence can reduce the amount of data of the residual image obtained subsequently.

[0014] In combination with the first aspect, in certain implementations of the first aspect, the first condition is that the similarity of image contents of at least two grid images among the multiple grid images is greater than or equal to a preset similarity.

[0015] The above-mentioned preset similarity can also be called a similarity threshold, and the size of the preset similarity can be flexibly set according to actual needs. For example, the above-mentioned preset similarity can be set to 50%. When the similarity of the image content of the two grid images is greater than or equal to 50%, the two grid images meet the first condition. The 50% here is only an example, and the preset similarity can also be set to 55%, 60%, 65%, etc. In short, any suitable size of preset similarity can be set according to needs.

[0016] In addition, the first condition may also be that the similarity between the image contents of at least two grid images among the plurality of grid images is greater than a preset similarity.

[0017] When the image sequence includes at least two grid images with similar image contents, the data volume of the residual image obtained subsequently can be reduced by performing inter-frame prediction on the image sequence.

[0018] In combination with the first aspect, in certain implementations of the first aspect, the first condition is that image contents of at least two grid images among the multiple grid images correspond to the same geographical area.

[0019] Since the similarity of images in the same geographical area is generally high, when the image sequence includes at least two grid images located in the same geographical area, the data volume of the residual image obtained subsequently can be smaller by performing inter-frame prediction on the image sequence.

[0020] The size of the geographic area may be a preset size, and the size of the geographic area may be set according to actual needs. For example, the size of the geographic area may be a 2m×2m area. It should be understood that the size of the geographic area of ​​2m×2m here is only an example, and the size of the geographic area may also be set to a 1m×1m area, a 3m×32m area, etc. In short, any suitable size of the geographic area may be set according to needs.

[0021] For another example, the size of the above-mentioned geographical area is an area of ​​2m×2m. When the two grid images are both located in the area of ​​2m×2m, the two grid images meet the first condition.

[0022] In combination with the first aspect, in certain implementations of the first aspect, at least two grid images among the plurality of grid images are grid images of adjacent frames.

[0023] That is to say, the at least two grid images satisfying the first condition are adjacent frames.

[0024] For example, grid image 1, grid image 2 and grid image 3 in the above image sequence meet the above first condition. Then, grid image 1, grid image 2 and grid image 3 can be the i-th frame, i+1-th frame and i+2-th frame in the image sequence, respectively. Where i is a positive integer.

[0025] In combination with the first aspect, in certain implementations of the first aspect, the pixel value of the grid image in the above-mentioned image sequence is represented by M bits, each bit has a value of 0 or 1, and M is an integer greater than 1. The above-mentioned method also includes: processing the grid image in the image sequence to obtain a processed grid image.

[0026] Among them, the value of the i-th bit among the N bits of the pixel value of the grid image in the image sequence is opposite to the value of the i-th bit among the N bits of the pixel value of the grid image in the processed image sequence, and the N bits of the pixel value of the grid image in the image sequence and the N bits of the pixel value of the grid image in the processed image sequence are in the same position.

[0027] The N bits of the pixel value of the grid image in the image sequence are located after and adjacent to the first bit of the pixel value of the grid image in the image sequence, the first bit is the bit with a value of 1 and the highest bit number in the pixel value of the grid image in the image sequence, i and N are both positive integers, i≤N, N<M.

[0028] In the present application, the encoding end can reduce the data volume of the grid image by inverting the pixel values ​​of the grid image when the pixel values ​​of the grid image contain continuous placeholders, thereby making the encoded code stream occupy less storage space, and the decoding end can restore the grid image processed (inverted) by the encoding end by (inverting) the decoded grid image, thereby obtaining the final grid image.

[0029] In combination with the first aspect, in certain implementations of the first aspect, the pixel value of the grid image in the above-mentioned image sequence is represented by M bits, each bit has a value of 0 or 1, and M is an integer greater than 1. The above-mentioned method also includes: processing the grid image in the image sequence to obtain a processed grid image.

[0030] Among them, the value of the first bit among the N bits of the grid image in the processed image sequence is opposite to the value of the first bit among the N bits of the grid image in the image sequence, and the value of the i+1th bit among the N bits of the grid image in the processed image sequence is the result of XOR processing of the value of the i+1th bit among the N bits of the grid image in the image sequence and the i-th bit among the N bits of the grid image in the processed image sequence.

[0031] The N bits of the pixel value of the grid image in the image sequence are located after and adjacent to the first bit of the pixel value of the grid image in the image sequence, and the first bit is the bit with a value of 1 and the highest bit number in the pixel value of the grid image in the image sequence, or, the N bits of the pixel value of the grid image in the image sequence are located after and adjacent to the highest bit of the pixel value of the grid image in the image sequence.

[0032] The N bits of the pixel values ​​of the grid image in the image sequence are in the same position as the N bits of the pixel values ​​of the grid image in the processed image sequence, the number of the i-th bit is higher than the number of the i+1-th bit, i and N are both positive integers, i≤N, N<M.

[0033] In the present application, the encoding end performs XOR processing on the pixel values ​​of the grid image, so as to reduce the data amount of the grid image when the pixel values ​​of the grid image contain continuous placeholders, thereby making the encoded code stream occupy less storage space, and the decoding end performs (XOR) processing on the decoded grid image, so as to restore the grid image processed (XOR processed) by the encoding end, and thus obtain the final grid image.

[0034] In combination with the first aspect, in certain implementations of the first aspect, the pixel value of the above-mentioned residual image is represented by M bits, each bit has a value of 0 or 1, and M is an integer greater than 1. The above-mentioned residual image of the grid image in the image sequence and the predicted image of the grid image in the image sequence obtain the grid image in the image sequence, specifically including: processing the residual image to obtain a processed residual image; obtaining the grid image in the image sequence according to the processed residual image and the predicted image of the grid image in the image sequence.

[0035] Among them, the value of the i-th bit among the N bits of the pixel value of the above-mentioned residual image is opposite to the value of the i-th bit among the N bits of the pixel value of the processed residual image, and the N bits of the pixel value of the residual image and the N bits of the pixel value of the processed residual image are in the same position.

[0036] The N bits of the pixel value of the residual image are located after the first bit of the pixel value of the residual image and are adjacent to the first bit. The first bit is the bit with a value of 1 and the highest bit number in the pixel value of the residual image. i and N are both positive integers, i≤N, N<M.

[0037] In the present application, the encoding end performs inversion processing on the pixel values ​​of the residual image, so that when the pixel values ​​of the residual image contain continuous placeholders, the data amount of the residual image can be reduced, thereby making the bit stream generated by the encoding occupy less storage space, and the decoding end performs (inversion) processing on the decoded residual image, so as to restore the residual image processed (inverted) by the encoding end, and then obtain the final image block according to the processed residual image.

[0038] In combination with the first aspect, in certain implementations of the first aspect, the pixel value of the above-mentioned residual image is represented by M bits, each bit has a value of 0 or 1, and M is an integer greater than 1. The above-mentioned residual image of the grid image in the image sequence and the predicted image of the grid image in the image sequence obtain the grid image in the image sequence, including: processing the residual image to obtain a processed residual image; obtaining the grid image in the image sequence according to the processed residual image and the predicted image of the grid image in the image sequence.

[0039] Among them, the value of the first bit among the N bits of the residual image after the above processing is opposite to the value of the first bit among the N bits of the residual image, and the value of the i+1th bit among the N bits of the residual image after the processing is the result of XOR processing of the value of the i+1th bit among the N bits of the residual image and the i-th bit among the N bits of the residual image after the processing.

[0040] The N bits of the pixel value of the residual image are located after the first bit of the pixel value of the residual image and are adjacent to the first bit, and the first bit is the bit with a value of 1 and the highest bit number in the pixel value of the residual image, or, the N bits of the pixel value of the residual image are located after the highest bit of the pixel value of the residual image and are adjacent to the highest bit of the pixel value of the residual image.

[0041] The N bits of the pixel value of the residual image are located at the same position as the N bits of the pixel value of the processed residual image, the number of bits of the i-th bit is higher than the number of bits of the i+1-th bit, i and N are both positive integers, i≤N, N<M;

[0042] In the present application, the encoder performs XOR processing on the pixel values ​​of the residual image, so that when the pixel values ​​of the residual image contain continuous placeholders, the data amount of the residual image can be reduced, thereby making the bitstream generated by the encoding occupy less storage space, and the decoder performs (XOR) processing on the decoded residual image, so as to restore the residual image processed (XOR processed) by the encoder, and then obtain the final image block according to the processed residual image.

[0043] In combination with the first aspect, in certain implementations of the first aspect, obtaining the residual image of the grid image in the image sequence based on the code stream includes: performing inverse transformation, inverse quantization and entropy decoding on the code stream to obtain the residual image.

[0044] The above decoding process corresponds to lossy encoding of the image, which can make the encoded code stream occupy as little storage space as possible.

[0045] In combination with the first aspect, in some implementations of the first aspect, the obtaining of the residual image of the grid image in the image sequence according to the bitstream includes: performing entropy decoding processing on the bitstream to obtain the residual image.

[0046] The above decoding method of only performing entropy decoding corresponds to lossless coding, which can avoid image distortion as much as possible and ensure the final display effect of the image.

[0047] In combination with the first aspect, in some implementations of the first aspect, the code stream is obtained by encoding the positioning layer.

[0048] In combination with the first aspect, in certain implementations of the first aspect, the pixel value of the positioning layer includes rasterized elevation data.

[0049] The pixel value of the positioning layer contains M bits, each bit representing the occupied position of each grid. Since the pixel value of the positioning layer contains the rasterized elevation data, and the probability of the character "11" appearing in the high position of the elevation data is relatively high, the pixel value of the positioning layer containing the elevation data is processed to reduce the data volume of the pixel value of the positioning layer, thereby reducing the storage space occupied by the final code stream.

[0050] In combination with the first aspect, in some implementations of the first aspect, the value of M is any one of 8, 10 and 12.

[0051] In a second aspect, an image encoding method is provided, the method comprising: acquiring an image sequence; performing inter-frame prediction on a grid image in the image sequence to obtain a residual image; and encoding the residual image to obtain an encoded bitstream.

[0052] The image sequence includes a plurality of grid images, and the geographical locations corresponding to the plurality of grid images are close or the image contents of the plurality of grid images are similar. Specifically, the geographical locations corresponding to the plurality of grid images are close or the image contents of the plurality of grid images are similar, which means that the plurality of grid images meet the first condition.

[0053] It should be understood that the above-mentioned inter-frame prediction of the grid image in the image sequence can be to first predict the grid image to obtain a predicted image of the grid image, and then subtract the grid image from the predicted image of the grid image to obtain a residual image of the grid image.

[0054] Specifically, the predicted image of the above-mentioned grid image can be composed of the predicted blocks of each image block in the grid image, and the residual image of the grid image can be composed of the residual blocks of each image block in the grid image. When encoding at the encoding end, the residual image can be divided into image blocks, and then the prediction block of each image block is obtained, and the residual block of each image block is obtained according to each image block and the residual block of each image block, thereby obtaining the residual image of the grid image.

[0055] In the present application, since the image sequence contains grid images with similar geographical locations or image contents, when inter-frame prediction is performed on the image sequence, the amount of data of the obtained residual image can be smaller, and thus the code stream obtained by encoding the image sequence finally occupies less storage space, which is convenient for storage or transmission.

[0056] In combination with the second aspect, in certain implementations of the second aspect, the first condition is that a distance between geographical locations corresponding to at least two grid images among the multiple grid images is less than a preset distance.

[0057] The above-mentioned preset distance may also be a distance threshold, and the size of the preset distance may be flexibly set according to actual needs.

[0058] For example, the preset distance can be set to 3m (3 meters). When the distance between the corresponding geographical locations of two grid images is less than 3m, the two grid images meet the first condition. The 3m here is just an example, and the preset distance can also be set to 5m and 6m, etc. In short, any suitable preset distance can be set as needed.

[0059] In addition, the first condition may also be that a distance between geographical locations corresponding to at least two grid images among the plurality of grid images is less than or equal to a preset distance.

[0060] Since images with similar geographical locations are generally highly similar, when an image sequence includes at least two grid images with corresponding geographical locations being adjacent, performing inter-frame prediction on the image sequence can reduce the amount of data of the residual image obtained subsequently.

[0061] In combination with the second aspect, in certain implementations of the second aspect, the first condition is that the similarity of image contents of at least two grid images among the multiple grid images is greater than or equal to a preset similarity.

[0062] When the image sequence includes at least two grid images with similar image contents, the data volume of the residual image obtained subsequently can be reduced by performing inter-frame prediction on the image sequence.

[0063] The above-mentioned preset similarity may also be referred to as a similarity threshold, and the value of the preset similarity may be flexibly set according to actual needs.

[0064] For example, the above-mentioned preset similarity can be set to 50%. When the similarity of the image contents of two grid images is greater than or equal to 50%, the two grid images meet the first condition. The 50% here is only an example, and the preset similarity can also be set to 55%, 60%, 65%, etc. In short, any suitable preset similarity can be set as needed.

[0065] In addition, the first condition may also be that the similarity between the image contents of at least two grid images among the plurality of grid images is greater than a preset similarity.

[0066] When the image sequence includes at least two grid images with similar image contents, the data volume of the residual image obtained subsequently can be reduced by performing inter-frame prediction on the image sequence.

[0067] In combination with the second aspect, in certain implementations of the second aspect, the first condition is that image contents of at least two grid images in the image sequence correspond to the same geographical area.

[0068] Since the similarity of images in the same geographical area is generally high, when the image sequence includes at least two grid images located in the same geographical area, the data volume of the residual image obtained subsequently can be smaller by performing inter-frame prediction on the image sequence.

[0069] The size of the geographic area may be a preset size, and the size of the geographic area may be set according to actual needs. For example, the size of the geographic area may be a 2m×2m area. It should be understood that the size of the geographic area of ​​2m×2m here is only an example, and the size of the geographic area may also be set to a 1m×1m area, a 3m×32m area, etc. In short, any suitable size of the geographic area may be set according to needs.

[0070] For another example, the size of the above-mentioned geographical area is an area of ​​2m×2m. When the two grid images are both located in the area of ​​2m×2m, the two grid images meet the first condition.

[0071] In combination with the second aspect, in certain implementations of the second aspect, at least two grid images among the plurality of grid images are grid images of adjacent frames.

[0072] It should be understood that the at least two grid images satisfying the first condition may be adjacent frames. For example, grid image 1, grid image 2, and grid image 3 in the above image sequence satisfy the above first condition. Then, grid image 1, grid image 2, and grid image 3 may be the i-th frame, i+1-th frame, and i+2-th frame in the image sequence, respectively. Where i is a positive integer.

[0073] In combination with the second aspect, in certain implementations of the second aspect, the pixel value of the grid image in the image sequence is represented by M bits, each bit has a value of 0 or 1, and M is an integer greater than 1. Before performing inter-frame prediction on the grid image in the image sequence, the above method also includes: processing the grid image in the image sequence to obtain a processed grid image; performing inter-frame prediction on the grid image in the image sequence, including: performing inter-frame prediction on the processed grid image to obtain a residual image.

[0074] Among them, the value of the i-th bit among the N bits of the pixel value of the grid image in the image sequence is opposite to the value of the i-th bit among the N bits of the pixel value of the grid image in the processed image sequence, and the N bits of the pixel value of the grid image in the image sequence and the N bits of the pixel value of the grid image in the processed image sequence are in the same position.

[0075] The N bits of the pixel value of the grid image in the image sequence are located after and adjacent to the first bit of the pixel value of the grid image in the image sequence, the first bit is the bit with a value of 1 and the highest bit number in the pixel value of the grid image in the image sequence, i and N are both positive integers, i≤N, N<M.

[0076] In the present application, the encoding end can reduce the data volume of the grid image by inverting the pixel values ​​of the grid image when the pixel values ​​of the grid image contain continuous placeholders, thereby making the bit stream generated by the encoding occupy less storage space.

[0077] In combination with the second aspect, in certain implementations of the second aspect, the pixel value of the grid image in the image sequence is represented by M bits, each bit has a value of 0 or 1, and M is an integer greater than 1. Before performing inter-frame prediction on the grid image in the image sequence, the above method also includes: processing the grid image in the image sequence to obtain a processed grid image; performing inter-frame prediction on the grid image in the image sequence, including: performing inter-frame prediction on the processed grid image to obtain a residual image.

[0078] Among them, the value of the first bit among the N bits of the grid image in the processed image sequence is opposite to the value of the first bit among the N bits of the grid image in the image sequence, and the value of the i+1th bit among the N bits of the grid image in the processed image sequence is the result of XOR processing of the value of the i+1th bit among the N bits of the grid image in the image sequence and the i-th bit among the N bits of the grid image in the processed image sequence.

[0079] The N bits of the pixel value of the grid image in the image sequence are located after and adjacent to the first bit of the pixel value of the grid image in the image sequence, and the first bit is the bit with a value of 1 and the highest bit number in the pixel value of the grid image in the image sequence, or, the N bits of the pixel value of the grid image in the image sequence are located after and adjacent to the highest bit of the pixel value of the grid image in the image sequence.

[0080] The N bits of the pixel values ​​of the grid image in the image sequence are in the same position as the N bits of the pixel values ​​of the grid image in the processed image sequence, the number of the i-th bit is higher than the number of the i+1-th bit, i and N are both positive integers, i≤N, N<M.

[0081] In the present application, the encoding end can reduce the data volume of the grid image by performing XOR processing on the pixel values ​​of the grid image when the pixel values ​​of the grid image contain continuous placeholders, thereby making the bit stream generated by the encoding occupy less storage space.

[0082] In combination with the second aspect, in certain implementations of the second aspect, the pixel value of the residual image is represented by M bits, each bit has a value of 0 or 1, and M is an integer greater than 1, and is characterized in that the above-mentioned encoding of the residual image to obtain a coded bitstream includes: processing the residual image to obtain a processed residual image; encoding the processed residual image to obtain a coded bitstream.

[0083] The value of the i-th bit in the N bits of the pixel value of the residual image is opposite to the value of the i-th bit in the N bits of the pixel value of the processed residual image, and the N bits of the pixel value of the residual image and the N bits of the pixel value of the processed residual image are located in the same position.

[0084] The N bits of the pixel value of the residual image are located after the first bit of the pixel value of the residual image and are adjacent to the first bit. The first bit is the bit with a value of 1 and the highest bit number in the pixel value of the residual image. i and N are both positive integers, i≤N, N<M.

[0085] In an embodiment of the present application, the encoding end can reduce the amount of data of the residual image by performing XOR processing on the pixel values ​​of the residual image when the pixel values ​​of the residual image contain continuous placeholders, thereby making the bitstream generated by the encoding occupy less storage space.

[0086] In combination with the second aspect, in certain implementations of the second aspect, the pixel value of the residual image is represented by M bits, each bit has a value of 0 or 1, and M is an integer greater than 1, and is characterized in that the above-mentioned encoding of the residual image to obtain a coded bitstream includes: processing the residual image to obtain a processed residual image; encoding the processed residual image to obtain a coded bitstream.

[0087] Among them, the value of the i-th bit in the N bits of the pixel value of the residual image is opposite to the value of the i-th bit in the N bits of the pixel value of the processed residual image, and the value of the i+1-th bit in the N bits of the processed residual image is the result of XOR processing of the value of the i+1-th bit in the N bits of the residual image and the i-th bit in the N bits of the processed residual image.

[0088] The N bits of the pixel value of the residual image are located after the first bit of the pixel value of the residual image and are adjacent to the first bit, and the first bit is the bit with a value of 1 and the highest bit number in the pixel value of the residual image, or, the N bits of the pixel value of the residual image are located after the highest bit of the pixel value of the residual image and are adjacent to the highest bit of the pixel value of the residual image.

[0089] The N bits of the pixel value of the residual image are located at the same position as the N bits of the pixel value of the processed residual image, the number of the i-th bit is higher than the number of the i+1-th bit, i and N are both positive integers, i≤N, N<M.

[0090] In an embodiment of the present application, the encoding end can reduce the amount of data of the residual image by performing XOR processing on the pixel values ​​of the residual image when the pixel values ​​of the residual image contain continuous placeholders, thereby making the bitstream generated by the encoding occupy less storage space.

[0091] In combination with the second aspect, in some implementations of the second aspect, encoding the residual image to obtain the encoded bitstream includes: transforming, quantizing and entropy encoding the residual image to obtain the encoded bitstream.

[0092] By adopting lossy coding, the encoded bit stream can occupy as little storage space as possible.

[0093] In combination with the second aspect, in some implementations of the second aspect, encoding the residual image to obtain the encoded bitstream includes: performing entropy encoding processing on the residual image to obtain the encoded bitstream.

[0094] By adopting lossless coding, image distortion can be avoided as much as possible and the final display effect of the image can be guaranteed.

[0095] In combination with the second aspect, in certain implementations of the second aspect, the grid images in the above-mentioned image sequence come from the positioning layer.

[0096] In combination with the second aspect, in certain implementations of the second aspect, the pixel value of the above-mentioned positioning layer includes rasterized elevation data.

[0097] The pixel value of the positioning layer contains M bits, each bit representing the occupied position of each grid. Since the pixel value of the positioning layer contains the rasterized elevation data, and the probability of the character "11" appearing in the high position of the elevation data is relatively high, the pixel value of the positioning layer containing the elevation data is processed to reduce the data volume of the pixel value of the positioning layer, thereby reducing the storage space occupied by the final code stream.

[0098] In combination with the second aspect, in some implementations of the second aspect, the value of M is any one of 8, 10 and 12.

[0099] According to a third aspect, an image decoding device is provided, the device comprising a module corresponding to the method of the first aspect, and the corresponding module can implement each step of the method of the first aspect.

[0100] According to a fourth aspect, an image encoding device is provided, which includes a module corresponding to the method of the second aspect, and the corresponding module can implement each step of the method of the second aspect.

[0101] The image decoding device in the third aspect or the image encoding device in the fourth aspect may include one or more modules, and any one of the one or more modules may be composed of any one of a circuit, a field programmable gate array FPGA, a special application integrated circuit ASIC and a general-purpose processor.

[0102] According to a fifth aspect, an image decoding device is provided, comprising a memory and a processor, wherein the processor calls a program code stored in the memory to execute the method according to the first aspect.

[0103] In a sixth aspect, an image encoding device is provided, comprising a memory and a processor, wherein the processor calls a program code stored in the memory to execute the method of the second aspect.

[0104] Optionally, the above-mentioned memory is a non-volatile memory.

[0105] Optionally, the memory and the processor are coupled to each other.

[0106] The above-mentioned image decoding device may also be referred to as an image decoder, and the above-mentioned image encoding device may also be referred to as an image encoder.

[0107] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and the instructions enable one or more processors to execute the method in the first aspect or the second aspect.

[0108] Any one of the one or more processors mentioned above may be composed of any one of a circuit, a field programmable gate array FPGA, an application specific integrated circuit ASIC and a general-purpose processor.

[0109] In an eighth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute part or all of the steps of the method in the first aspect or the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] Figure 1 is a schematic block diagram of an example of a video encoding system for implementing an embodiment of the present application;

[0111] Figure 2 is a schematic structural block diagram of an example of a video encoder for implementing an embodiment of the present application;

[0112] Figure 3 is a schematic structural block diagram of an example of a video decoder for implementing an embodiment of the present application;

[0113] Figure 4 is a schematic structural block diagram of an example of a video decoding system for implementing an embodiment of the present application;

[0114] Figure 5 is a schematic structural block diagram of an example of a video decoding device for implementing an embodiment of the present application;

[0115] Figure 6 is a schematic block diagram of an example of an encoding device or a decoding device for implementing an embodiment of the present application;

[0116] Figure 7 is a schematic diagram of the spatial and temporal candidate motion information of the current coding unit;

[0117] Figure 8 is a schematic diagram of the process of the image encoding method according to an embodiment of the present application;

[0118] Fig. 9 is a schematic diagram of the process of the image encoding method according to an embodiment of the present application;

[0119] Fig.10 is a schematic diagram of the process of the image encoding method according to an embodiment of the present application;

[0120] Fig.11 is a schematic flow chart of an image decoding method according to an embodiment of the present application;

[0121] Fig.12 is a schematic flow chart of an image decoding method according to an embodiment of the present application;

[0122] Fig.13 is a schematic flow chart of an image decoding method according to an embodiment of the present application;

[0123] Fig.14 is a schematic flow chart of an image encoding method according to an embodiment of the present application;

[0124] Fig.15 is a schematic flow chart of an image encoding method according to an embodiment of the present application;

[0125] Fig.16 It is a schematic diagram of the encoding performance corresponding to the inversion operation;

[0126] Fig.17 It is a schematic diagram of the encoding performance corresponding to the XOR operation;

[0127] Fig.18 is a schematic flow chart of an image encoding method according to an embodiment of the present application;

[0128] Fig.19 It is a schematic diagram of an image sequence including I frames and P frames;

[0129] Fig. 20 It is a schematic diagram of converting 3D point cloud data into 2D plane data;

[0130] Fig.21 is a schematic block diagram of an image decoding device according to an embodiment of the present application;

[0131] Fig. 22 is a schematic block diagram of an image encoding device according to an embodiment of the present application;

[0132] Fig.23 is a schematic block diagram of an image decoding device according to an embodiment of the present application;

[0133] Fig.24 It is a schematic block diagram of an image encoding device according to an embodiment of the present application. DETAILED DESCRIPTION

[0134] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0135] In the following description, reference will be made to the drawings that form a part of the present application and illustrate specific aspects of the embodiments of the present application or specific aspects of the embodiments of the present application that can be used by way of illustration. It should be understood that the embodiments of the present application can also be used in other aspects and may include structural or logical changes not depicted in the drawings. Therefore, the following detailed description should not be understood in a restrictive sense, and the scope of the present application should be defined by the appended claims.

[0136] For example, it should be understood that the disclosure in conjunction with a described method may also be applicable to a corresponding device or system that performs the method, and vice versa.

[0137] For another example, if one or more specific method steps are described, the corresponding device may include one or more units, such as functional units, to perform the described one or more method steps (for example, one unit performs one or more steps, or multiple units, each of which performs one or more of the multiple steps), even if such one or more units are not explicitly described or illustrated in the drawings.

[0138] In addition, if a specific device is described based on one or more units such as functional units, the corresponding method may include a step to perform the function of one or more units (e.g., one step performs the function of one or more units, or multiple steps, each of which performs the function of one or more units in multiple units), even if such one or more steps are not explicitly described or illustrated in the drawings. Further, it should be understood that unless otherwise explicitly stated, the features of the various exemplary embodiments and / or aspects described herein can be combined with each other.

[0139] The technical solutions involved in the embodiments of the present application can be applied to the H.266 standard and future video coding standards. The terms used in the implementation method of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application. The following is a brief introduction to some concepts that may be involved in the embodiments of the present application.

[0140] The image encoding and decoding in this application are the same as the encoding and decoding of video images. Some basic processes and related contents of the encoding and decoding of video images are introduced in detail below.

[0141] Video coding generally refers to processing a sequence of pictures that form a video or video sequence. In the field of video coding, the terms "picture", "frame" or "image" can be used as synonyms. Video coding used in this article means video encoding or video decoding. Video encoding is performed on the source side and generally includes processing (e.g., by compression) the original video picture to reduce the amount of data required to represent the video picture, thereby more efficiently storing and / or transmitting. Video decoding is performed on the destination side and generally includes inverse processing relative to the encoder to reconstruct the video picture. The "encoding" of video pictures involved in the embodiments should be understood as involving "encoding" or "decoding" of video sequences. The combination of the encoding part and the decoding part is also referred to as codec (encoding and decoding).

[0142] A video sequence includes a series of pictures, which are further divided into slices, and slices are further divided into blocks. Video coding is performed in blocks. In some new video coding standards, the concept of blocks is further expanded. For example, in the H.264 standard, there are macroblocks (MB), which can be further divided into multiple prediction blocks (partitions) that can be used for predictive coding. In the high-performance video coding (HEVC) standard, basic concepts such as coding units (CU), prediction units (PU) and transform units (TU) are used to functionally divide various block units and describe them using a new tree-based structure. For example, CU can be divided into smaller CUs according to a quadtree, and smaller CUs can be further divided to form a quadtree structure. CU is the basic unit for dividing and encoding coded images. There are similar tree structures for PU and TU. PU can correspond to prediction blocks and is the basic unit of predictive coding. CU is further divided into multiple PUs according to the division mode. TU may correspond to a transform block, which is a basic unit for transforming a prediction residual. However, no matter CU, PU or TU, they all belong to the concept of a block (or image block) in essence.

[0143] For example, in HEVC, a CTU is split into multiple CUs using a quadtree structure represented as a coding tree. A decision is made at the CU level whether to use inter-picture (temporal) or intra-picture (spatial) prediction to encode a picture area. Each CU can be further split into one, two, or four PUs according to the PU split type. The same prediction process is applied within a PU, and relevant information is transmitted to the decoder on a PU basis. After obtaining the residual block by applying the prediction process based on the PU split type, the CU can be divided into transform units (TUs) according to other quadtree structures similar to the coding tree for the CU. In the latest developments in video compression technology, quad-tree and binarytree (QTBT) are used to split frames to split coding blocks. In the QTBT block structure, the CU can be square or rectangular in shape.

[0144] In this document, for ease of description and understanding, the image block to be encoded in the current encoded image may be referred to as the current image block. For example, in encoding, it refers to the block currently being encoded; in decoding, it refers to the block currently being decoded. The decoded image block in the reference image used to predict the current image block is referred to as a reference block, that is, the reference block is a block that provides a reference signal for the current image block, wherein the reference signal represents the pixel value in the image block. The block in the reference image that provides a prediction signal for the current image block may be referred to as a prediction block, wherein the prediction signal represents the pixel value or sampling value or sampling signal in the prediction block. For example, after traversing multiple reference blocks, the best reference block is found. This best reference block will provide a prediction for the current image block, and this block is referred to as a prediction block.

[0145] In the case of lossless video coding, the original video picture can be reconstructed, that is, the reconstructed video picture has the same quality as the original video picture (assuming no transmission loss or other data loss during storage or transmission). In the case of lossy video coding, further compression is performed by, for example, quantization to reduce the amount of data required to represent the video picture, but the decoder side cannot fully reconstruct the video picture, that is, the quality of the reconstructed video picture is lower or worse than the quality of the original video picture.

[0146] Several video coding standards of H.261 belong to the category of "lossy hybrid video codecs" (i.e., combining spatial and temporal prediction in the sample domain with 2D transform coding in the transform domain for applying quantization). Each picture of a video sequence is usually divided into a set of non-overlapping blocks, which are usually encoded at the block level. In other words, the encoder side usually processes, i.e., encodes the video at the block (video block) level, for example, by generating a prediction block through spatial (intra-picture) prediction and temporal (inter-picture) prediction, subtracting the prediction block from the current image block (currently processed or to be processed block) to obtain a residual block, transforming the residual block in the transform domain and quantizing the residual block to reduce the amount of data to be transmitted (compressed), while the decoder side applies the inverse processing part relative to the encoder to the coded or compressed block to reconstruct the current image block for representation. In addition, the encoder replicates the decoder processing cycle, so that the encoder and decoder generate the same prediction (e.g., intra-frame prediction and inter-frame prediction) and / or reconstruction for processing, i.e., encoding subsequent blocks.

[0147] The following describes the system architecture used in the embodiments of the present application. Figure 1 , Figure 1 A schematic block diagram of a video encoding and decoding system 10 used in an embodiment of the present application is given as an example. Figure 1 As shown, the video encoding and decoding system 10 may include a source device 12 and a destination device 14. The source device 12 generates encoded video data, and therefore, the source device 12 may be referred to as a video encoding device. The destination device 14 may decode the encoded video data generated by the source device 12, and therefore, the destination device 14 may be referred to as a video decoding device. Various implementations of the source device 12, the destination device 14, or both may include one or more processors and a memory coupled to the one or more processors. The memory may include, but is not limited to, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), a flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures that can be accessed by a computer, as described herein. Source device 12 and destination device 14 may include a variety of devices, including desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called "smart" phones, televisions, cameras, display devices, digital media players, video game consoles, in-vehicle computers, wireless communication devices, or the like.

[0148] Although Figure 1The source device 12 and the destination device 14 are illustrated as separate devices, but device embodiments may also include both the source device 12 and the destination device 14 or the functions of both, that is, the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions. In such embodiments, the source device 12 or the corresponding functionality and the destination device 14 or the corresponding functionality may be implemented using the same hardware and / or software, or using separate hardware and / or software, or any combination thereof.

[0149] The source device 12 and the destination device 14 may be connected in communication via a link 13, and the destination device 14 may receive the encoded video data from the source device 12 via the link 13. The link 13 may include one or more media or devices capable of moving the encoded video data from the source device 12 to the destination device 14. In one example, the link 13 may include one or more communication media that enable the source device 12 to transmit the encoded video data directly to the destination device 14 in real time. In this example, the source device 12 may modulate the encoded video data according to a communication standard (e.g., a wireless communication protocol), and may transmit the modulated video data to the destination device 14. The one or more communication media may include wireless and / or wired communication media, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, such as a local area network, a wide area network, or a global network (e.g., the Internet). The one or more communication media may include routers, switches, base stations, or other devices that facilitate communication from the source device 12 to the destination device 14.

[0150] The source device 12 includes an encoder 20. Optionally, the source device 12 may also include a picture source 16, a picture preprocessor 18, and a communication interface 22. In a specific implementation, the encoder 20, the picture source 16, the picture preprocessor 18, and the communication interface 22 may be hardware components in the source device 12, or may be software programs in the source device 12. They are described as follows:

[0151] The picture source 16 may include or may be any type of picture capture device, for example, for capturing real-world pictures, and / or any type of pictures or comments (for screen content encoding, some text on the screen is also considered to be part of the picture or image to be encoded) generating device, for example, a computer graphics processor for generating computer animation pictures, or any type of device for acquiring and / or providing real-world pictures, computer animation pictures (e.g., screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). The picture source 16 may be a camera for capturing pictures or a memory for storing pictures, and the picture source 16 may also include any type of (internal or external) interface for storing previously captured or generated pictures and / or acquiring or receiving pictures. When the picture source 16 is a camera, the picture source 16 may be, for example, a local or integrated camera integrated in the source device; when the picture source 16 is a memory, the picture source 16 may be, for example, a local or integrated memory integrated in the source device. When the picture source 16 includes an interface, the interface may be, for example, an external interface for receiving pictures from an external video source, such as an external picture capture device, such as a camera, an external memory, or an external picture generating device, such as an external computer graphics processor, a computer, or a server. The interface may be any type of interface according to any proprietary or standardized interface protocol, such as a wired or wireless interface, an optical interface.

[0152] Among them, the picture can be regarded as a two-dimensional array or matrix of pixels (picture element). The pixels in the array can also be called sampling points. The number of sampling points in the array or picture in the horizontal and vertical directions (or axes) defines the size and / or resolution of the picture. In order to represent the color, three color components are usually used, that is, the picture can be represented as or include three sampling arrays. For example, in the RBG format or color space, the picture includes corresponding red, green and blue sampling arrays. However, in video coding, each pixel is usually represented in a brightness / chroma format or color space, such as for a picture in the YUV format, including a brightness component indicated by Y (sometimes also indicated by L) and two chroma components indicated by U and V. The brightness (luma) component Y represents the brightness or grayscale level intensity (for example, the two are the same in a grayscale picture), and the two chroma (chroma) components U and V represent the chroma or color information components. Accordingly, the picture in the YUV format includes a brightness sampling array of brightness sampling values ​​(Y), and two chroma sampling arrays of chroma values ​​(U and V). An image in RGB format may be converted or transformed into YUV format, and vice versa, which process is also referred to as color conversion or transformation. If the image is black and white, the image may only include a brightness sampling array. In the embodiment of the present application, the image transmitted from the image source 16 to the image processor may also be referred to as raw image data 17.

[0153] The image preprocessor 18 is used to receive the original image data 17 and perform preprocessing on the original image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. For example, the preprocessing performed by the image preprocessor 18 may include refurbishment, color format conversion (e.g., from RGB format to YUV format), color adjustment, or denoising.

[0154] The encoder 20 (or video encoder 20) is used to receive the pre-processed picture data 19, and process the pre-processed picture data 19 using a relevant prediction mode (such as the prediction mode in each embodiment of this invention), so as to provide encoded picture data 21 (hereinafter further based on Figure 2 or Figure 4 or Figure 5 ). In some embodiments, the encoder 20 may be used to execute various embodiments described below to implement the application of the encoding method described in this application on the encoding side.

[0155] The communication interface 22 can be used to receive the encoded picture data 21 and transmit the encoded picture data 21 to the destination device 14 or any other device (such as a memory) through the link 13 for storage or direct reconstruction. The other device can be any device for decoding or storage. The communication interface 22 can be used, for example, to encapsulate the encoded picture data 21 into a suitable format, such as a data packet, for transmission on the link 13.

[0156] The destination device 14 includes a decoder 30. Optionally, the destination device 14 may also include a communication interface 28, a picture post-processor 32, and a display device 34. They are described as follows:

[0157] The communication interface 28 can be used to receive the encoded picture data 21 from the source device 12 or any other source, such as a storage device, such as an encoded picture data storage device. The communication interface 28 can be used to transmit or receive the encoded picture data 21 via the link 13 between the source device 12 and the destination device 14 or via any type of network, such as a direct wired or wireless connection, any type of network such as a wired or wireless network or any combination thereof, or any type of private network and public network, or any combination thereof. The communication interface 28 can be used, for example, to decapsulate the data packets transmitted by the communication interface 22 to obtain the encoded picture data 21.

[0158] Both communication interface 28 and communication interface 22 may be configured as unidirectional communication interfaces or bidirectional communication interfaces and may be used, for example, to send and receive messages to establish connections, confirm and exchange any other information related to communication links and / or data transmissions such as encoded picture data transmissions.

[0159] The decoder 30 (or decoder 30) is configured to receive the encoded picture data 21 and provide decoded picture data 31 or decoded picture 31 (hereinafter further described based on Figure 3 or Figure 4 or Figure 5 ). In some embodiments, the decoder 30 may be used to execute various embodiments described below to implement the decoding method described in this application on the decoding side.

[0160] The picture post-processor 32 is used to perform post-processing on the decoded picture data 31 (also called reconstructed picture data) to obtain post-processed picture data 33. The post-processing performed by the picture post-processor 32 may include: color format conversion (for example, from YUV format to RGB format), color adjustment, repair or resampling, or any other processing, and can also be used to transmit the post-processed picture data 33 to the display device 34.

[0161] A display device 34 is provided for receiving the post-processed picture data 33 to display the picture to, for example, a user or viewer. The display device 34 may be or may include any type of display for presenting the reconstructed picture, such as an integrated or external display or monitor. For example, the display may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0162] Although, Figure 1 1 and 14 are shown as separate devices, but device embodiments may also include both the source device 12 and the destination device 14 or both functionalities, that is, the source device 12 or the corresponding functionality and the destination device 14 or the corresponding functionality. In such embodiments, the source device 12 or the corresponding functionality and the destination device 14 or the corresponding functionality may be implemented using the same hardware and / or software, or using separate hardware and / or software, or any combination thereof.

[0163] It is obvious to a person skilled in the art based on the description that the functionality or Figure 1 The presence and (exact) division of the functionality of the source device 12 and / or destination device 14 shown may vary depending on the actual device and application. The source device 12 and the destination device 14 may include any of a variety of devices, including any type of handheld or stationary device, such as a notebook or laptop computer, a mobile phone, a smart phone, a tablet or tablet computer, a video camera, a desktop computer, a set-top box, a television, a camera, a car device, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or a content distribution server), a broadcast receiver device, a broadcast transmitter device, etc., and may not use or use any type of operating system.

[0164] The encoder 20 and the decoder 30 may be implemented as any of a variety of suitable circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof. If the technology is partially implemented in software, the device may store the instructions of the software in a suitable non-transitory computer-readable storage medium, and may use one or more processors to execute the instructions in hardware to perform the technology of the present application. Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) may be considered to be one or more processors.

[0165] In some cases, Figure 1 The video encoding and decoding system 10 shown in is only an example, and the technology of the present application can be applied to video encoding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between encoding and decoding devices. In other examples, data can be retrieved from local storage, streamed over a network, etc. A video encoding device can encode data and store the data in a memory, and / or a video decoding device can retrieve data from a memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other but only encode data to a memory and / or retrieve data from a memory and decode data.

[0166] See also Figure 2 , Figure 2 Schematic / conceptual block diagram of an example of an encoder 20 for implementing an embodiment of the present application is shown. Figure 2 In the example of , the encoder 20 includes a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a buffer 216, a loop filter unit 220, a decoded picture buffer (decoded picture buffer, DPB) 230, a prediction processing unit 260 and an entropy coding unit 270. The prediction processing unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254 and a mode selection unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown in the figure). Figure 2 The illustrated encoder 20 may also be referred to as a hybrid video encoder or a video encoder according to a hybrid video codec.

[0167] For example, the residual calculation unit 204, the transform processing unit 206, the quantization unit 208, the prediction processing unit 260, and the entropy coding unit 270 form a forward signal path of the encoder 20, while for example, the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the buffer 216, the loop filter 220, the decoded picture buffer (DPB) 230, and the prediction processing unit 260 form a backward signal path of the encoder, wherein the backward signal path of the encoder corresponds to the signal path of the decoder (see Figure 3 Decoder 30 in ).

[0168] The encoder 20 receives a picture 201 or an image block 203 of the picture 201, e.g., a picture in a sequence of pictures forming a video or a video sequence, via, e.g., an input 202. The image block 203 may also be referred to as a current picture block or a picture block to be encoded, and the picture 201 may be referred to as a current picture or a picture to be encoded (particularly when the current picture is distinguished from other pictures in video coding, e.g., previously encoded and / or decoded pictures in the same video sequence, i.e., a video sequence that also includes the current picture).

[0169] An embodiment of the encoder 20 may include a segmentation unit ( Figure 2 201 into a plurality of blocks, such as image blocks 203, typically into a plurality of non-overlapping blocks. The segmentation unit may be used to use the same block size for all pictures in the video sequence and define a corresponding grid of block sizes, or to change the block size between pictures or subsets or groups of pictures and to segment each picture into corresponding blocks.

[0170] In one example, prediction processing unit 260 of encoder 20 may be used to perform any combination of the above-described partitioning techniques.

[0171] Like the picture 201, the image block 203 is also or can be regarded as a two-dimensional array or matrix of sampling points with sample values, although its size is smaller than that of the picture 201. In other words, the image block 203 may include, for example, one sampling array (e.g., a luminance array in the case of the black and white picture 201) or three sampling arrays (e.g., one luminance array and two chrominance arrays in the case of a color picture) or any other number and / or type of arrays depending on the color format applied. The number of sampling points in the horizontal and vertical directions (or axes) of the image block 203 defines the size of the image block 203.

[0172] like Figure 2 The encoder 20 shown is used to encode the picture 201 block by block, for example, encoding and prediction are performed on each image block 203.

[0173] The residual calculation unit 204 is used to calculate the residual block 205 based on the picture image block 203 and the prediction block 265 (other details of the prediction block 265 are provided below), for example, by subtracting the sample value of the prediction block 265 from the sample value of the picture image block 203 sample by sample (pixel by pixel) to obtain the residual block 205 in the sample domain.

[0174] The transform processing unit 206 is used to apply a transform such as discrete cosine transform (DCT) or discrete sine transform (DST) on the sample values ​​of the residual block 205 to obtain a transform coefficient 207 in the transform domain. The transform coefficient 207 can also be called a transform residual coefficient and represents the residual block 205 in the transform domain.

[0175] The transform processing unit 206 can be used to apply an integer approximation of the DCT / DST, such as the transform specified for HEVC / H.265. Compared to the orthogonal DCT transform, this integer approximation is usually scaled by a certain factor. In order to maintain the norm of the residual block processed by the forward transform and the inverse transform, an additional scaling factor is applied as part of the transform process. The scaling factor is usually selected based on certain constraints, such as the scaling factor is a power of 2 for the shift operation, the bit depth of the transform coefficients, the trade-off between accuracy and implementation cost, etc. For example, a specific scaling factor is specified for the inverse transform on the decoder 30 side by, for example, the inverse transform processing unit 212 (and for the corresponding inverse transform on the encoder 20 side by, for example, the inverse transform processing unit 212), and accordingly, a corresponding scaling factor can be specified for the forward transform on the encoder 20 side by the transform processing unit 206.

[0176] The quantization unit 208 is used to quantize the transform coefficients 207, for example, by applying scalar quantization or vector quantization to obtain quantized transform coefficients 209. The quantized transform coefficients 209 may also be referred to as quantized residual coefficients 209. The quantization process may reduce the bit depth associated with some or all of the transform coefficients 207. For example, an n-bit transform coefficient may be rounded down to an m-bit transform coefficient during quantization, where n is greater than m. The degree of quantization may be modified by adjusting a quantization parameter (QP). For example, for scalar quantization, different scales may be applied to achieve finer or coarser quantization. A smaller quantization step size corresponds to finer quantization, while a larger quantization step size corresponds to coarser quantization. A suitable quantization step size may be indicated by a quantization parameter (QP). For example, the quantization parameter may be an index to a predefined set of suitable quantization step sizes. For example, a smaller quantization parameter may correspond to fine quantization (a smaller quantization step size), a larger quantization parameter may correspond to coarse quantization (a larger quantization step size), and vice versa. Quantization may include division by a quantization step size and corresponding quantization or inverse quantization, for example, performed by inverse quantization 210, or may include multiplication by a quantization step size. Embodiments according to some standards such as HEVC may use a quantization parameter to determine the quantization step size. In general, the quantization step size may be calculated based on the quantization parameter using a fixed-point approximation of an equation containing division. Additional scaling factors may be introduced for quantization and inverse quantization to recover the norm of the residual block that may be modified due to the scale used in the fixed-point approximation of the equation for the quantization step size and the quantization parameter. In an example embodiment, the scales of the inverse transform and inverse quantization may be merged. Alternatively, a custom quantization table may be used and sent from an encoder to a decoder through a signal, for example in a bitstream. Quantization is a lossy operation, where the larger the quantization step size, the greater the loss.

[0177] The inverse quantization unit 210 is used to apply the inverse quantization of the quantization unit 208 on the quantized coefficients to obtain inverse quantized coefficients 211, for example, applying an inverse quantization scheme of the quantization scheme applied by the quantization unit 208 based on or using the same quantization step size as the quantization unit 208. The inverse quantized coefficients 211 may also be referred to as inverse quantized residual coefficients 211, corresponding to the transform coefficients 207, although the loss due to quantization is generally different from that of the transform coefficients.

[0178] The inverse transform processing unit 212 is used to apply the inverse transform of the transform applied by the transform processing unit 206, such as an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST), to obtain an inverse transform block 213 in the sample domain. The inverse transform block 213 may also be referred to as an inverse transform dequantized block 213 or an inverse transform residual block 213.

[0179] The reconstruction unit 214 (e.g., the summer 214) is used to add the inverse transform block 213 (i.e., the reconstructed residual block 213) to the prediction block 265 to obtain the reconstructed block 215 in the sample domain, for example, by adding the sample values ​​of the reconstructed residual block 213 to the sample values ​​of the prediction block 265.

[0180] Optionally, the buffer unit 216 of the line buffer 216 (or simply "buffer" 216) can be used to buffer or store the reconstructed blocks 215 and the corresponding sample values ​​for, for example, intra-frame prediction. In other embodiments, the encoder can be used to use the unfiltered reconstructed blocks and / or the corresponding sample values ​​stored in the buffer unit 216 for any type of estimation and / or prediction, such as intra-frame prediction.

[0181] For example, embodiments of the encoder 20 may be configured such that the buffer unit 216 is used not only to store the reconstructed blocks 215 used for the intra prediction 254, but also for the loop filter unit 220 (in Figure 2 ), and / or, for example, such that the buffer unit 216 and the decoded picture buffer unit 230 form a buffer. Other embodiments may be used to store the filtered block 221 and / or the blocks or samples ( Figure 2 (not shown) is used as input or basis for intra-frame prediction 254.

[0182] The loop filter unit 220 (or simply loop filter 220) is used to filter the reconstructed block 215 to obtain a filtered block 221, so as to smoothly perform pixel conversion or improve video quality. The loop filter unit 220 is intended to represent one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or other filters, such as a bilateral filter, an adaptive loop filter (ALF), or a sharpening or smoothing filter, or a collaborative filter. Although the loop filter unit 220 is in Figure 2 2 is shown as an in-loop filter, but in other configurations, the loop filter unit 220 may be implemented as a post-loop filter. The filtered block 221 may also be referred to as a filtered reconstructed block 221. The decoded picture buffer 230 may store the reconstructed coding block after the loop filter unit 220 performs a filtering operation on the reconstructed coding block.

[0183] Embodiments of the encoder 20 (correspondingly, the loop filter unit 220) can be used to output loop filter parameters (e.g., sample adaptive offset information), for example, directly or after entropy encoding by the entropy encoding unit 270 or any other entropy encoding unit, such that the decoder 30 can receive and apply the same loop filter parameters for decoding.

[0184] The decoded picture buffer (DPB) 230 may be a reference picture memory for storing reference picture data for use by the encoder 20 in encoding video data. The DPB 230 may be formed by any of a variety of memory devices, such as a dynamic random access memory (DRAM) (including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM)) or other types of memory devices. The DPB 230 and the buffer 216 may be provided by the same memory device or separate memory devices. In one example, the decoded picture buffer (DPB) 230 is used to store the filtered block 221. The decoded picture buffer 230 may further be used to store other previously filtered blocks of the same current picture or a different picture, such as a previously reconstructed picture, such as the previously reconstructed and filtered block 221, and may provide a complete previously reconstructed, i.e., decoded picture (and corresponding reference blocks and samples) and / or a portion of the reconstructed current picture (and corresponding reference blocks and samples), such as for inter-frame prediction. In one example, if the reconstructed block 215 is reconstructed without in-loop filtering, the decoded picture buffer (DPB) 230 is used to store the reconstructed block 215.

[0185] The prediction processing unit 260, also referred to as the block prediction processing unit 260, is used to receive or obtain an image block 203 (a current image block 203 of a current picture 201) and reconstructed picture data, such as reference samples of the same (current) picture from a buffer 216 and / or reference picture data 231 of one or more previously decoded pictures from a decoded picture buffer 230, and to process such data for prediction, i.e. to provide a prediction block 265 which may be an inter-prediction block 245 or an intra-prediction block 255.

[0186] The mode selection unit 262 may be used to select a prediction mode (eg, intra or inter prediction mode) and / or a corresponding prediction block 245 or 255 for use as a prediction block 265 to calculate the residual block 205 and reconstruct the reconstructed block 215 .

[0187] Embodiments of the mode selection unit 262 may be used to select a prediction mode (e.g., from those prediction modes supported by the prediction processing unit 260) that provides the best match or minimum residual (minimum residual means better compression in transmission or storage), or provides minimum signaling overhead (minimum signaling overhead means better compression in transmission or storage), or considers or balances both. The mode selection unit 262 may be used to determine the prediction mode based on rate distortion optimization (RDO), i.e., select a prediction mode that provides minimum rate distortion optimization, or select a prediction mode whose associated rate distortion at least satisfies the prediction mode selection criteria.

[0188] The prediction processing performed by an example of encoder 20 (eg, by prediction processing unit 260) and the mode selection performed (eg, by mode selection unit 262) will be explained in detail below.

[0189] As described above, the encoder 20 is used to determine or select the best or optimal prediction mode from a (predetermined) prediction mode set. The prediction mode set may include, for example, an intra-frame prediction mode and / or an inter-frame prediction mode.

[0190] The intra-frame prediction mode set may include 35 different intra-frame prediction modes, for example, non-directional modes such as DC (or mean) mode and planar mode, or directional modes as defined in H.265, or may include 67 different intra-frame prediction modes, for example, non-directional modes such as DC (or mean) mode and planar mode, or directional modes as defined in H.266 under development.

[0191] In a possible implementation, the inter-frame prediction mode set depends on the available reference picture (i.e., at least part of the decoded picture stored in the DBP 230 as described above) and other inter-frame prediction parameters, such as whether to use the entire reference picture or only a part of the reference picture, such as a search window area around the area of ​​the current image block, to search for the best matching reference block, and / or, for example, whether to apply pixel interpolation such as half-pixel and / or quarter-pixel interpolation. The inter-frame prediction mode set may include, for example, an advanced motion vector prediction (AMVP) mode and a merge mode. In a specific implementation, the inter-frame prediction mode set may include an improved control point-based AMVP mode according to an embodiment of the present application, and an improved control point-based merge mode. In one example, the intra-frame prediction unit 254 may be used to perform any combination of the inter-frame prediction techniques described below.

[0192] In addition to the above prediction modes, embodiments of the present application may also apply skip mode and / or direct mode.

[0193] The prediction processing unit 260 can be further used to partition the image block 203 into smaller block partitions or sub-blocks, for example, by iteratively using quad-tree (QT) partitioning, binary-tree (BT) partitioning or triple-tree (TT) partitioning, or any combination thereof, and for performing prediction for each of the block partitions or sub-blocks, for example, wherein the mode selection includes selecting a tree structure of the partitioned image block 203 and selecting a prediction mode to be applied to each of the block partitions or sub-blocks.

[0194] The inter-frame prediction unit 244 may include a motion estimation (ME) unit ( Figure 2 ) and a motion compensation (MC) unit ( Figure 2 ). The motion estimation unit is used to receive or obtain a picture image block 203 (a current picture image block 203 of a current picture 201) and a decoded picture 231, or at least one or more previously reconstructed blocks, for example, reconstructed blocks of one or more other / different previously decoded pictures 231, to perform motion estimation. For example, a video sequence may include a current picture and a previously decoded picture 31, or in other words, the current picture and the previously decoded picture 31 may be part of a picture sequence forming a video sequence, or form the picture sequence.

[0195] For example, the encoder 20 may be configured to select a reference block from a plurality of reference blocks of the same or different pictures in a plurality of other pictures and to provide the reference block to the motion estimation unit ( Figure 2(not shown) provides a reference picture and / or provides an offset (spatial offset) between the position (X, Y coordinates) of a reference block and the position of a current image block as an inter-frame prediction parameter. The offset is also called a motion vector (MV).

[0196] The motion compensation unit is used to obtain inter-frame prediction parameters, and perform inter-frame prediction based on or using the inter-frame prediction parameters to obtain the inter-frame prediction block 245. Figure 2 Motion compensation performed by the motion compensation unit 246 (not shown) may include fetching or generating a prediction block based on a motion / block vector determined by motion estimation (possibly performing interpolation to sub-pixel accuracy). Interpolation filtering may produce additional pixel samples from known pixel samples, thereby potentially increasing the number of candidate prediction blocks that may be used to encode the picture block. Upon receiving the motion vector for the PU of the current picture block, the motion compensation unit 246 may locate the prediction block to which the motion vector points in a reference picture list. The motion compensation unit 246 may also generate syntax elements associated with blocks and video slices for use by the decoder 30 when decoding picture blocks of the video slice.

[0197] Specifically, the inter-frame prediction unit 244 may transmit syntax elements to the entropy coding unit 270, wherein the syntax elements include inter-frame prediction parameters (such as indication information of selecting an inter-frame prediction mode for current image block prediction after traversing multiple inter-frame prediction modes). In possible application scenarios, if there is only one inter-frame prediction mode, the inter-frame prediction parameters may not be carried in the syntax elements, and the decoding end 30 may directly use the default prediction mode for decoding. It is understandable that the inter-frame prediction unit 244 may be used to perform any combination of inter-frame prediction techniques.

[0198] The intra prediction unit 254 is used to obtain, for example, receive a picture block 203 (current picture block) of the same picture and one or more previously reconstructed blocks, for example, reconstructed adjacent blocks, for intra estimation. For example, the encoder 20 can be used to select an intra prediction mode from a plurality of (predetermined) intra prediction modes.

[0199] Embodiments of the encoder 20 may be configured to select an intra prediction mode based on an optimization criterion, such as minimum residual (eg, the intra prediction mode that provides a prediction block 255 most similar to the current picture block 203) or minimum rate distortion.

[0200] The intra prediction unit 254 is further configured to determine an intra prediction block 255 based on the intra prediction parameters of the selected intra prediction mode. In any case, after selecting the intra prediction mode for the block, the intra prediction unit 254 is further configured to provide the intra prediction parameters, i.e., provide information indicating the selected intra prediction mode for the block, to the entropy encoding unit 270. In one example, the intra prediction unit 254 can be configured to perform any combination of intra prediction techniques.

[0201] Specifically, the intra prediction unit 254 may transmit syntax elements to the entropy coding unit 270, wherein the syntax elements include intra prediction parameters (such as indication information of selecting an intra prediction mode for current image block prediction after traversing multiple intra prediction modes). In a possible application scenario, if there is only one intra prediction mode, the intra prediction parameters may not be carried in the syntax elements, and the decoding end 30 may directly use the default prediction mode for decoding.

[0202] The entropy coding unit 270 is used to apply an entropy coding algorithm or scheme (e.g., a variable length coding (VLC) scheme, a context adaptive VLC (CAVLC) scheme, an arithmetic coding scheme, a context adaptive binary arithmetic coding (CABAC), a syntax-based context-adaptive binary arithmetic coding (SBAC), a probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to a single or all (or none) of the quantized residual coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, and / or loop filter parameters to obtain the encoded picture data 21 that can be output through the output 272 in the form of, for example, an encoded bitstream 21. The encoded bitstream can be transmitted to the video decoder 30, or archived for later transmission or retrieval by the video decoder 30. The entropy coding unit 270 can also be used to entropy encode other syntax elements of the current video slice being encoded.

[0203] Other structural variations of the video encoder 20 may be used to encode the video stream. For example, a non-transform based encoder 20 may directly quantize the residual signal without a transform processing unit 206 for certain blocks or frames. In another embodiment, the encoder 20 may have a quantization unit 208 and an inverse quantization unit 210 combined into a single unit.

[0204] Specifically, in the embodiment of the present application, the encoder 20 may be used to implement the video encoding process described in the embodiments below.

[0205] It should be understood that the video encoder in the present application may include only some modules in the video encoder 20. For example, the video encoder in the present application may include an image decoding unit and a partitioning unit. The image decoding unit may be composed of one or more units of an entropy decoding unit, a prediction unit, an inverse transform unit, and an inverse quantization unit.

[0206] In addition, other structural changes of the video encoder 20 can be used to encode the video stream. For example, for some image blocks or image frames, the video encoder 20 can directly quantize the residual signal without processing by the transform processing unit 206, and accordingly, it does not need to be processed by the inverse transform processing unit 212; or, for some image blocks or image frames, the video encoder 20 does not generate residual data, and accordingly, it does not need to be processed by the transform processing unit 206, the quantization unit 208, the inverse quantization unit 210 and the inverse transform processing unit 212; or, the video encoder 20 can directly store the reconstructed image block as a reference block without processing by the filter 220; or, the quantization unit 208 and the inverse quantization unit 210 in the video encoder 20 can be combined together. The loop filter 220 is optional, and in the case of lossless compression encoding, the transform processing unit 206, the quantization unit 208, the inverse quantization unit 210 and the inverse transform processing unit 212 are optional. It should be understood that according to different application scenarios, the inter-frame prediction unit 244 and the intra-frame prediction unit 254 can be selectively enabled.

[0207] See also Figure 3 , Figure 3 A schematic / conceptual block diagram of an example of a decoder 30 for implementing an embodiment of the present application is shown. The video decoder 30 is used to receive encoded picture data (e.g., encoded bitstream) 21, for example, encoded by the encoder 20, to obtain a decoded picture 231. During the decoding process, the video decoder 30 receives video data from the video encoder 20, such as an encoded video bitstream representing picture blocks of an encoded video slice and associated syntax elements.

[0208] exist Figure 3 In the example of , decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., summer 314), a buffer 316, a loop filter 320, a decoded picture buffer 330, and a prediction processing unit 360. Prediction processing unit 360 may include an inter-prediction unit 344, an intra-prediction unit 354, and a mode selection unit 362. In some examples, video decoder 30 may perform substantially the same as reference. Figure 2 The encoding pass described by video encoder 20 is the inverse of the decoding pass.

[0209] The entropy decoding unit 304 is used to perform entropy decoding on the encoded picture data 21 to obtain, for example, quantized coefficients 309 and / or decoded encoding parameters ( Figure 3 The entropy decoding unit 304 is further configured to forward the inter-prediction parameters, intra-prediction parameters, and / or other syntax elements to the prediction processing unit 360. The video decoder 30 may receive syntax elements at the video slice level and / or the video block level.

[0210] The inverse quantization unit 310 may be functionally the same as the inverse quantization unit 110, the inverse transform processing unit 312 may be functionally the same as the inverse transform processing unit 212, the reconstruction unit 314 may be functionally the same as the reconstruction unit 214, the buffer 316 may be functionally the same as the buffer 216, the loop filter 320 may be functionally the same as the loop filter 220, and the decoded picture buffer 330 may be functionally the same as the decoded picture buffer 230.

[0211] The prediction processing unit 360 may include an inter-prediction unit 344 and an intra-prediction unit 354, wherein the inter-prediction unit 344 may be functionally similar to the inter-prediction unit 244, and the intra-prediction unit 354 may be functionally similar to the intra-prediction unit 254. The prediction processing unit 360 is generally used to perform block prediction and / or obtain a prediction block 365 from the encoded data 21, and receive or obtain prediction related parameters and / or information about the selected prediction mode from, for example, the entropy decoding unit 304 (explicitly or implicitly).

[0212] When the video slice is encoded as an intra-coded (I) slice, the intra-prediction unit 354 of the prediction processing unit 360 is used to generate a prediction block 365 for a picture block of the current video slice based on the signaled intra-prediction mode and data from a previously decoded block of the current frame or picture. When the video frame is encoded as an inter-coded (i.e., B or P) slice, the inter-prediction unit 344 (e.g., motion compensation unit) of the prediction processing unit 360 is used to generate a prediction block 365 for the video block of the current video slice based on the motion vector and other syntax elements received from the entropy decoding unit 304. For inter-prediction, the prediction block may be generated from one of the reference pictures within one of the reference picture lists. Video decoder 30 may construct reference frame lists: List 0 and List 1 using a default construction technique based on the reference pictures stored in DPB 330.

[0213] The prediction processing unit 360 is configured to determine prediction information for a video block of a current video slice by parsing motion vectors and other syntax elements, and to generate a prediction block for the current video block being decoded using the prediction information. In one example of the present application, the prediction processing unit 360 uses some of the received syntax elements to determine a prediction mode (e.g., intra-frame or inter-frame prediction) for encoding a video block of a video slice, an inter-frame prediction slice type (e.g., a B slice, a P slice, or a GPB slice), construction information for one or more of a reference picture list for the slice, a motion vector for each inter-frame coded video block of the slice, an inter-frame prediction state for each inter-frame coded video block of the slice, and other information to decode the video block of the current video slice. In another example of the present application, the syntax elements received by the video decoder 30 from the bitstream include syntax elements received in one or more of an adaptive parameter set (APS), a sequence parameter set (SPS), a picture parameter set (PPS), or a slice header.

[0214] Inverse quantization unit 310 may be used to inverse quantize (i.e., dequantize) quantized transform coefficients provided in the bitstream and decoded by entropy decoding unit 304. The inverse quantization process may include using quantization parameters calculated by video encoder 20 for each video block in a video slice to determine a degree of quantization that should be applied and likewise a degree of inverse quantization that should be applied.

[0215] The inverse transform processing unit 312 is used to apply an inverse transform (eg, an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients to generate a residual block in the pixel domain.

[0216] The reconstruction unit 314 (e.g., the summer 314) is used to add the inverse transform block 313 (i.e., the reconstructed residual block 313) to the prediction block 365 to obtain the reconstructed block 315 in the sample domain, for example, by adding the sample values ​​of the reconstructed residual block 313 to the sample values ​​of the prediction block 365.

[0217] The loop filter unit 320 is used to filter the reconstructed block 315 (during or after the encoding cycle) to obtain a filtered block 321, thereby smoothing pixel conversion or improving video quality. In one example, the loop filter unit 320 can be used to perform any combination of the filtering techniques described below. The loop filter unit 320 is intended to represent one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or other filters, such as a bilateral filter, an adaptive loop filter (ALF), or a sharpening or smoothing filter, or a collaborative filter. Although the loop filter unit 320 is in Figure 3 3. Although shown as an in-loop filter in FIG. 3, in other configurations, the loop filter unit 320 may be implemented as a post-loop filter.

[0218] The decoded video blocks 321 in a given frame or picture are then stored in a decoded picture buffer 330 which stores reference pictures for subsequent motion compensation.

[0219] Decoder 30 is operable to output decoded picture 31 , eg, via output 332 , for presentation to or viewing by a user.

[0220] Other variations of the video decoder 30 may be used to decode the compressed bitstream. For example, the decoder 30 may generate an output video stream without the loop filter unit 320. For example, a non-transform based decoder 30 may directly inverse quantize the residual signal without the inverse transform processing unit 312 for certain blocks or frames. In another embodiment, the video decoder 30 may have the inverse quantization unit 310 and the inverse transform processing unit 312 combined into a single unit.

[0221] Specifically, in the embodiment of the present application, the decoder 30 is used to implement the video decoding method described in the embodiments below.

[0222] It should be understood that the video encoder in the present application may include only some modules in the video encoder 30. For example, the video encoder in the present application may include a partition unit and an image encoding unit. The image encoding unit may be composed of one or more units of a prediction unit, a transform unit, a quantization unit, and an entropy encoding unit.

[0223] In addition, other structural changes of the video decoder 30 can be used to decode the encoded video bitstream. For example, the video decoder 30 can generate an output video stream without being processed by the filter 320; or, for some image blocks or image frames, the entropy decoding unit 304 of the video decoder 30 does not decode the quantized coefficients, and accordingly does not need to be processed by the inverse quantization unit 310 and the inverse transform processing unit 312. The loop filter 320 is optional; and for lossless compression, the inverse quantization unit 310 and the inverse transform processing unit 312 are optional. It should be understood that, according to different application scenarios, the inter-frame prediction unit and the intra-frame prediction unit can be selectively enabled.

[0224] It should be understood that in the encoder 20 and decoder 30 of the present application, the processing result of a certain link can be further processed and output to the next link. For example, after the interpolation filtering, motion vector derivation or loop filtering links, the processing result of the corresponding link can be further clamped (clip) or shifted (shift) and other operations.

[0225] For example, the motion vector of the control point of the current image block derived from the motion vector of the adjacent affine coding block (the coding block predicted by the affine motion model can be called an affine coding block), or the motion vector of the sub-block of the current image block derived, can be further processed, and this application does not limit this. For example, the value range of the motion vector is constrained so that it is within a certain bit width. Assuming that the bit width of the allowed motion vector is bitDepth, the range of the motion vector is -2^(bitDepth-1)~2^(bitDepth-1)-1, where the "^" symbol represents the power. If bitDepth is 16, the value range is -32768~32767. If bitDepth is 18, the value range is -131072~131071.

[0226] For example, the value of the motion vector (for example, the motion vector MV of four 4x4 sub-blocks in an 8x8 image block) can also be constrained so that the maximum difference between the integer parts of the four 4x4 sub-blocks MV does not exceed N (for example, N can be 1) pixels.

[0227] See also Figure 4 , Figure 4 According to an exemplary embodiment, Figure 2 The encoder 20 and / or Figure 330. The video decoding system 40 may implement a combination of various techniques of the embodiments of the present application. In the illustrated embodiment, the video decoding system 40 may include an imaging device 41, an encoder 20, a decoder 30 (and / or a video codec implemented by logic circuits 47 of a processing unit 46), an antenna 42, one or more processors 43, one or more memories 44, and / or a display device 45.

[0228] like Figure 4 As shown, imaging device 41, antenna 42, processing unit 46, logic circuit 47, encoder 20, decoder 30, processor 43, memory 44 and / or display device 45 can communicate with each other. As discussed, although video decoding system 40 is illustrated with encoder 20 and decoder 30, in different examples, video decoding system 40 may include only encoder 20 or only decoder 30.

[0229] In some examples, antenna 42 can be used to transmit or receive a coded bit stream of video data. In addition, in some examples, display device 45 can be used to present video data. In some examples, logic circuit 47 can be implemented by processing unit 46. Processing unit 46 can include application-specific integrated circuit (ASIC) logic, graphics processor, general processor, etc. Video decoding system 40 can also include optional processor 43, which can similarly include application-specific integrated circuit (ASIC) logic, graphics processor, general processor, etc. In some examples, logic circuit 47 can be implemented by hardware, such as video encoding dedicated hardware, etc., and processor 43 can be implemented by general software, operating system, etc. In addition, memory 44 can be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.). In a non-limiting example, memory 44 can be implemented by cache memory. In some examples, logic circuitry 47 may access memory 44 (eg, for implementing an image buffer). In other examples, logic circuitry 47 and / or processing unit 46 may include memory (eg, cache, etc.) for implementing an image buffer, etc.

[0230] In some examples, encoder 20 implemented by logic circuits may include an image buffer (e.g., implemented by processing unit 46 or memory 44) and a graphics processing unit (e.g., implemented by processing unit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include encoder 20 implemented by logic circuits 47 to implement reference Figure 2 The various modules discussed in connection with the encoder and / or any other encoder system or subsystem described herein. Logic circuits may be used to perform the various operations discussed herein.

[0231] In some examples, decoder 30 may be implemented in a similar manner by logic circuit 47 to implement reference Figure 3 The decoder 30 and / or any other decoder system or subsystem described herein may include various modules discussed above. In some examples, the decoder 30 implemented by logic circuits may include an image buffer (implemented by processing unit 2820 or memory 44) and a graphics processing unit (implemented, for example, by processing unit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the decoder 30 implemented by logic circuits 47 to implement the reference Figure 3 and / or the various modules discussed above for any other decoder system or subsystem described herein.

[0232] In some examples, antenna 42 may be used to receive an encoded bitstream of video data. As discussed, the encoded bitstream may include data related to the encoded video frames, indicators, index values, mode selection data, etc. discussed herein, such as data related to the encoded partitions (e.g., transform coefficients or quantized transform coefficients, (as discussed) optional indicators, and / or data defining the encoded partitions). Video decoding system 40 may also include decoder 30 coupled to antenna 42 and used to decode the encoded bitstream. Display device 45 is used to present the video frames.

[0233] It should be understood that in the embodiments of the present application, for the examples described with reference to encoder 20, decoder 30 can be used to perform the reverse process. With respect to signaling syntax elements, decoder 30 can be used to receive and parse such syntax elements and decode the associated video data accordingly. In some examples, encoder 20 can entropy encode the syntax elements into an encoded video bitstream. In such examples, decoder 30 can parse such syntax elements and decode the associated video data accordingly.

[0234] See also Figure 5 , Figure 5FIG. 4 is a schematic diagram of the structure of a video decoding device 400 (eg, a video encoding device 400 or a video decoding device 400) provided in an embodiment of the present application. The video decoding device 400 is suitable for implementing the embodiments described herein. In one embodiment, the video decoding device 400 may be a video decoder (eg, Figure 3 decoder 30) or a video encoder (e.g. Figure 2 In another embodiment, the video decoding device 400 may be the above-mentioned Figure 3 Decoder 30 or Figure 2 One or more components in encoder 20.

[0235] The video decoding device 400 includes: an inlet port 410 and a receiving unit (Rx) 420 for receiving data, a processor, a logic unit or a central processing unit (CPU) 430 for processing data, a transmitter unit (Tx) 440 and an outlet port 450 for transmitting data, and a memory 460 for storing data. The video decoding device 400 may also include an optical-to-electrical conversion component and an electro-optical (EO) component coupled to the inlet port 410, the receiver unit 420, the transmitter unit 440 and the outlet port 450 for the outlet or inlet of an optical signal or an electrical signal.

[0236] The processor 430 is implemented by hardware and software. The processor 430 can be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 430 communicates with the inlet port 410, the receiver unit 420, the transmitter unit 440, the outlet port 450, and the memory 460. The processor 430 includes a decoding module 470 (e.g., an encoding module 470 or a decoding module 470). The encoding / decoding module 470 implements the embodiments disclosed herein to implement the encoding method / decoding method of the embodiment of the present application. For example, the encoding / decoding module 470 implements, processes, or provides various encoding operations. Therefore, the encoding / decoding module 470 provides substantial improvements to the functions of the video decoding device 400 and affects the conversion of the video decoding device 400 to different states. Alternatively, the encoding / decoding module 470 is implemented with instructions stored in the memory 460 and executed by the processor 430.

[0237] The memory 460 includes one or more disks, tape drives, and solid-state drives, and can be used as an overflow data storage device for storing programs when these programs are selectively executed, and for storing instructions and data read during program execution. The memory 460 can be volatile and / or non-volatile, and can be a read-only memory (ROM), a random access memory (RAM), a ternary content-addressable memory (TCAM), and / or a static random access memory (SRAM).

[0238] See also Figure 6 , Figure 6 According to an exemplary embodiment, it can be used as Figure 1 The simplified block diagram of the apparatus 500 of any one or both of the source device 12 and the destination device 14 in FIG. The apparatus 500 can implement the encoding method or the decoding method of the embodiment of the present application. In other words, Figure 6 A schematic block diagram of an implementation method of an encoding device or decoding device (referred to as a decoding device 500) of an embodiment of the present application. The decoding device 500 may include a processor 510, a memory 530, and a bus system 550. The processor and the memory are connected via a bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory. The memory of the decoding device stores program code, and the processor can call the program code stored in the memory to execute various video encoding or decoding methods described in this application, especially various new image block division methods. To avoid repetition, it is not described in detail here.

[0239] In the embodiment of the present application, the processor 510 may be a central processing unit (CPU), and the processor 510 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0240] The memory 530 may include a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may also be used as the memory 530. The memory 530 may include code and data 531 accessed by the processor 510 using the bus 550. The memory 530 may further include an operating system 533 and an application 535, which includes at least one program that allows the processor 510 to perform the video encoding or decoding method described in the present application. For example, the application 535 may include applications 1 to N, which further include a video encoding or decoding application (referred to as a video decoding application) that executes the video encoding or decoding method described in the present application.

[0241] The bus system 550 may include, in addition to the data bus, a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, various buses are labeled as the bus system 550 in the figure.

[0242] Optionally, the decoding device 500 may further include one or more output devices, such as a display 570. In one example, the display 570 may be a touch-sensitive display that combines a display with a touch-sensitive unit operable to sense touch input. The display 570 may be connected to the processor 510 via the bus 550.

[0243] In order to better understand the image prediction process in the encoding method and decoding method in the embodiments of the present application, some relevant concepts and basic contents of inter-frame prediction are first introduced in detail below.

[0244] Inter-frame prediction refers to finding a matching reference block for the current image block in the current image in the reconstructed image, and using the pixel values ​​of the pixels in the reference block as the predicted values ​​of the pixel values ​​of the pixels in the current image block (this process is called motion estimation (ME)).

[0245] Motion estimation is to try multiple reference blocks in the reference image for the current image block, and then use rate-distortion optimization (RDO) or other methods to finally determine one or two reference blocks from the multiple reference blocks (two reference blocks are required for bidirectional prediction), and use the reference blocks to perform inter-frame prediction on the current image block.

[0246] The motion information of the current image block includes the indication information of the prediction direction (usually forward prediction, backward prediction or bidirectional prediction), one or two motion vectors (MV) pointing to the reference block, and the indication information of the image where the reference block is located (usually represented by a reference frame index).

[0247] Forward prediction means that the current image block selects a reference image from the forward reference image set to obtain a reference block. Backward prediction means that the current image block selects a reference image from the backward reference image set to obtain a reference block. Bidirectional prediction means that a reference image is selected from each of the forward reference image set and the backward reference image set to obtain a reference block. When the bidirectional prediction method is used, there will be two reference blocks in the current coding block, each of which requires a motion vector and a reference frame index to indicate it, and then the predicted value of the pixel value in the current image block is determined based on the pixel values ​​in the two reference blocks.

[0248] In HEVC, there are two inter-frame prediction modes, namely AMVP mode and merge mode.

[0249] In the AMVP mode, the coded blocks adjacent to the current coding block in the spatial or temporal domain (referred to as neighboring blocks) are first traversed, and a candidate motion vector list is constructed according to the motion information of each neighboring block. Then, the optimal motion vector is determined from the candidate motion information list through the rate-distortion cost, and the candidate motion information with the smallest rate-distortion cost is used as the motion vector predictor (MVP) of the current coding block.

[0250] Among them, the positions of neighboring blocks and their traversal order are predefined. The rate-distortion cost can be calculated according to formula (1), where J is the rate-distortion cost, SAD is the sum of absolute differences (SAD) between the pixel prediction value obtained after motion estimation using the candidate motion vector prediction value and the original pixel value, R is the bit rate, λ is the Lagrange multiplier, and the encoder passes the index value of the selected motion vector prediction value in the candidate motion vector list and the reference frame index value to the decoder. Furthermore, the encoder can perform motion search in the neighborhood centered on the MVP to obtain the actual motion vector of the current coding block, and then pass the difference (motion vector difference) between the MVP and the actual motion vector to the decoder.

[0251] J=SAD+λR (1)

[0252] In addition, according to different motion models, the AMVP mode can be divided into an AMVP mode based on a translational model and an AMVP mode based on a non-translational model.

[0253] In merge mode, a candidate motion information list is first constructed through the motion information of the coded units adjacent to the current coding unit in the spatial domain or temporal domain, and then the optimal motion information is determined from the candidate motion information list through the rate-distortion cost as the motion information of the current coding unit. Finally, the index value of the position of the optimal motion information in the candidate motion information list (denoted as merge index, the same below) is passed to the decoding end.

[0254] In merge mode, the spatial and temporal candidate motion information of the current coding unit can be Figure 7 As shown, the spatial candidate motion information comes from five spatially adjacent blocks (A0, A1, B0, B1 and B2). If the adjacent block is not available or the prediction mode is intra-frame prediction, the adjacent block will not be added to the candidate motion information list.

[0255] The temporal candidate motion information of the current coding unit can be obtained by scaling the MV of the corresponding position block in the reference frame according to the picture order count (POC) of the reference frame and the current frame. When obtaining the corresponding position block of the reference frame, it can be determined whether the block at position T in the reference frame is available. If not, the block at position C is selected.

[0256] When the translation model is used for prediction, all pixels in the coding unit use the same motion information, and then motion compensation is performed based on the motion information to obtain the predicted value of the pixel of the coding unit. However, in the real world, there are many kinds of motion, and there are many objects with non-translational motion, such as rotating objects, roller coasters rotating in different directions, fireworks, and some stunts in movies, especially moving objects in user generated content (UGC) scenes. If the block motion compensation technology based on the translation motion model in the current coding standard is used for their encoding, the coding efficiency will be greatly affected. Therefore, in order to improve the coding effect, a prediction based on the non-translational motion model is proposed.

[0257] Non-translational motion model prediction refers to using the same motion model at the codec end to derive the motion information of each sub-motion compensation unit in the current coding block, and then performing motion compensation based on the motion information of the sub-motion compensation unit to obtain the predicted sub-block of each sub-block, thereby improving the prediction efficiency. Commonly used non-translational motion models include 4-parameter affine motion models or 6-parameter affine motion models.

[0258] Generally speaking, near the same geographical location, there are often the same or similar topographic features. Therefore, for the positioning layer, there is often a certain correlation (such as similar textures) between multiple grid images near the same geographical location in the positioning layer. If, when encoding the positioning layer, the multiple grid images near the same geographical location in the positioning layer are combined into an image sequence, and then the image sequence is inter-frame prediction encoded, the amount of data of the residual image of the grid image obtained according to the inter-frame prediction process can be made smaller, and the amount of data of the code stream generated by the final encoding is also relatively small, thereby reducing the storage overhead occupied by the code stream generated by the encoding.

[0259] The image encoding method and the image decoding method of the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0260] Figure 8 It is a schematic diagram of the process of the image encoding method according to an embodiment of the present application.

[0261] Figure 8 The method shown may be executed by an encoding device or an encoder. Figure 8 The main process of the image encoding method of the embodiment of the present application is shown. Figure 8 In the method shown, images with similar geographical locations can be grouped into an image sequence (or images with similar image contents can be grouped into an image sequence), and then encoded using an inter-frame prediction-based encoding method to obtain a bit stream. The inter-frame prediction-based encoding here refers to an encoding method that uses an inter-frame prediction method when predicting an image.

[0262] Since the image contents of geographically adjacent images are highly similar, inter-frame prediction is performed on the images in the image sequence to obtain a predicted image, and a residual image of the image is obtained based on the predicted image. This makes the data volume of the residual image obtained in the end smaller, and the data volume of the code stream generated in the final encoding is also smaller, thereby reducing the storage space occupied by the encoded code stream.

[0263] In order to further reduce the storage space occupied by the coded stream, an XOR process or an inversion process may be performed before or during the coding of the images in the image sequence, and then the images in the image sequence are encoded. This can reduce the amount of image data when the pixel values ​​of the images in the image sequence contain continuous placeholders, thereby making the coded stream occupy less storage space.

[0264] Specifically, the pixel values ​​of the image may be XORed or inverted after the image is obtained, or images with adjacent geographical locations may be first organized into an image sequence after the image is obtained, and then the images in the image sequence may be XORed or inverted.

[0265] like Fig. 9 As shown, after acquiring the image, the image is first subjected to XOR processing or inversion processing, and then images with close geographical locations are selected from the processed images to form an image sequence, and then the image sequence is encoded based on inter-frame prediction coding to obtain a code stream.

[0266] like Fig.10 As shown, after acquiring the image, the images with similar geographical locations are first grouped into an image sequence, and then the images in the image sequence are subjected to XOR processing or inversion processing, and then the processed image sequence is encoded based on the inter-frame prediction encoding method to obtain a code stream.

[0267] Let's combine Fig.11 The image decoding method of the embodiment of the present application is introduced in detail.

[0268] Fig.11 It is a schematic flowchart of the image decoding method of an embodiment of the present application. Fig.11 The method shown may be executed by a decoding device or a decoder. Fig.11 The method shown includes steps 1001 to 1004, and steps 1001 to 1004 are described in detail below.

[0269] 1001. Obtain code stream.

[0270] The code stream obtained in step 1001 may be as follows: Fig.14 The code stream obtained by encoding in the encoding method shown.

[0271] 1002. Obtain a residual image of a grid image in an image sequence according to a code stream.

[0272] In step 1002, the residual block of the current image block can be obtained by parsing the bitstream. For the specific parsing process, see Figure 3 The decoding process is shown.

[0273] The image sequence includes a plurality of grid images, and the plurality of grid images correspond to similar geographical locations or have similar image contents. Specifically, the plurality of grid images correspond to similar geographical locations or have similar image contents, which means that the plurality of grid images meet the first condition.

[0274] The first condition mentioned above can be expressed in many different specific forms.

[0275] Optionally, the first condition may be that a distance between geographical locations corresponding to at least two grid images in the plurality of grid images in the image sequence is less than a preset distance.

[0276] The above-mentioned preset distance may also be a distance threshold, and the size of the preset distance may be flexibly set according to actual needs. For example, the above-mentioned preset distance may be set to 3m (3 meters), and when the distance between the corresponding geographical locations of two grid images is less than 3m, the two grid images meet the first condition. The 3m here is only an example, and the preset distance may also be set to 5m and 6m, etc. In short, any suitable size of preset distance may be set according to needs.

[0277] In addition, the first condition may also be that a distance between geographical locations corresponding to at least two grid images among the plurality of grid images is less than or equal to a preset distance.

[0278] Since images with similar geographical locations are generally highly similar, when an image sequence includes at least two grid images with corresponding geographical locations being adjacent, performing inter-frame prediction on the image sequence can reduce the amount of data of the residual image obtained subsequently.

[0279] Optionally, the first condition may be that the similarity of image contents of at least two grid images in the image sequence is greater than or equal to a preset similarity.

[0280] The above-mentioned preset similarity can also be called a similarity threshold, and the size of the preset similarity can be flexibly set according to actual needs. For example, the above-mentioned preset similarity can be set to 50%. When the similarity of the image content of the two grid images is greater than or equal to 50%, the two grid images meet the first condition. The 50% here is only an example, and the preset similarity can also be set to 55%, 60%, 65%, etc. In short, any suitable size of preset similarity can be set according to needs.

[0281] In addition, the first condition may also be that the similarity between the image contents of at least two grid images among the plurality of grid images is greater than a preset similarity.

[0282] When the image sequence includes at least two grid images with similar image contents, the data volume of the residual image obtained subsequently can be reduced by performing inter-frame prediction on the image sequence.

[0283] Optionally, the first condition may be that image contents of at least two grid images among the plurality of grid images correspond to the same geographical area.

[0284] Since the similarity of images in the same geographical area is generally high, when the image sequence includes at least two grid images located in the same geographical area, the data volume of the residual image obtained subsequently can be smaller by performing inter-frame prediction on the image sequence.

[0285] The size of the geographic area may be a preset size, and the size of the geographic area may be set according to actual needs. For example, the size of the geographic area may be a 2m×2m area. It should be understood that the size of the geographic area of ​​2m×2m here is only an example, and the size of the geographic area may also be set to a 1m×1m area, a 3m×32m area, etc. In short, any suitable size of the geographic area may be set according to needs.

[0286] For another example, the size of the above-mentioned geographical area is an area of ​​2m×2m. When the two grid images are both located in the area of ​​2m×2m, the two grid images meet the first condition.

[0287] Furthermore, at least two grid images among the above-mentioned multiple grid images may be grid images of adjacent frames.

[0288] That is to say, at least two grid images satisfying the first condition are adjacent frames to each other. For example, grid image 1, grid image 2 and grid image 3 in the above image sequence satisfy the above first condition. Then, grid image 1, grid image 2 and grid image 3 may be the i-th frame, i+1-th frame and i+2-th frame in the image sequence, respectively. Where i is a positive integer.

[0289] 1003. Perform inter-frame prediction on the grid images in the image sequence to obtain predicted images of the grid images in the image sequence.

[0290] The process of obtaining the predicted image of the grid image in the image sequence through inter-frame prediction in step 1003 can be seen in Figure 3 The decoding process is shown.

[0291] 1004. Obtain a grid image in the image sequence according to a residual image of the grid image in the image sequence and a predicted image of the grid image in the image sequence.

[0292] In step 1004, the grid image in the image sequence may be obtained by superimposing the residual image of the grid image in the image sequence and the predicted image of the grid image in the image sequence. Fig.11 In the method shown, the prediction image of the grid image may be composed of the prediction blocks of each image block in the grid image, and the residual image of the grid image may be composed of the residual blocks of each image block in the grid image. Fig.11In the method, in the process of parsing the code stream to finally obtain the grid image in the image sequence, the decoding operation can be performed with the image block as the basic unit. Specifically, in step 1002, the residual block of the image block of the grid image in the image sequence can be obtained according to the code stream, and then in step 1003, the image block is inter-frame predicted to obtain the predicted block of the image block. Next, in step 1004, the corresponding image block is finally obtained according to the residual block and the predicted block. After obtaining each image block, the image blocks can be spliced ​​into a grid image, so that at the decoding end, it is equivalent to obtaining the grid image according to the residual image of the grid image and the predicted image of the grid image.

[0293] The specific implementation of the above step 1004 can be found in Figure 3 The decoding process is shown.

[0294] In the present application, since the image sequence contains grid images with similar geographical locations or image contents, when inter-frame prediction is performed on the image sequence, the amount of data of the obtained residual image can be smaller, thereby making the storage space occupied by the code stream corresponding to the final image sequence smaller, which is convenient for storage or transmission.

[0295] In an embodiment of the present application, before or during encoding of a grid image in an image sequence, the encoding end may also perform an inversion process or an XOR process on the pixel values ​​of the grid image in the image sequence to further reduce the storage space occupied by the code stream generated by the encoding, and correspondingly, during or after the decoding process, the decoding end may also perform an inversion process or an XOR process on the corresponding image to obtain the final image. This processing method is described in detail below.

[0296] Specifically, in Fig.11 In the method shown, after the grid images of the image sequence are decoded, the grid images in the image sequence may be processed to obtain the final grid images.

[0297] like Fig.12 As shown, after the grid image of the image sequence is obtained by decoding, Fig.11 The method shown also includes:

[0298] 1005. Process the grid image in the image sequence to obtain a processed grid image.

[0299] The pixel value of the grid image in the above image sequence is represented by M bits, each bit has a value of 0 or 1, and M is an integer greater than 1.

[0300] The processed grid image and the grid image obtained in the above step 1005 may satisfy the first relationship.

[0301] First relationship: the pixel values ​​of the processed grid image are opposite to the pixel values ​​of the grid image.

[0302] Specifically, under the first relationship, the value of the i-th bit among the N bits of the pixel value of the grid image in the image sequence is opposite to the value of the i-th bit among the N bits of the pixel value of the grid image in the processed image sequence, and the N bits of the pixel value of the grid image in the image sequence and the N bits of the pixel value of the grid image in the processed image sequence are located in the same position; the N bits of the pixel value of the grid image in the image sequence are located after the first bit of the pixel value of the grid image in the image sequence and are adjacent to the first bit, the first bit is the bit with the highest bit number and the value of 1 in the pixel value of the grid image in the image sequence, i and N are both positive integers, i≤N, N<M.

[0303] In the present application, the encoding end can reduce the data volume of the grid image by inverting the pixel values ​​of the grid image when the pixel values ​​of the grid image contain continuous placeholders, thereby making the encoded code stream occupy less storage space, and the decoding end can restore the grid image processed (inverted) by the encoding end by (inverting) the decoded grid image, thereby obtaining the final grid image.

[0304] After the grid images of the image sequence are decoded and obtained, in addition to performing an inversion process on the pixel values ​​of the grid images of the image sequence, an exclusive-OR process may also be performed on the pixel values ​​of the grid images of the image sequence.

[0305] Therefore, the processed grid image and the grid image obtained in the above step 1005 may satisfy not only the first relationship but also the second relationship.

[0306] Second relationship: The pixel value of the processed grid image is the XOR result of the pixel values.

[0307] Specifically, under the second relationship, the value of the first bit among the N bits of the grid image in the processed image sequence is opposite to the value of the first bit among the N bits of the grid image in the image sequence, and the value of the i+1th bit among the N bits of the grid image in the processed image sequence is the result of exclusive-OR processing of the value of the i+1th bit among the N bits of the grid image in the image sequence and the i-th bit among the N bits of the grid image in the processed image sequence; the N bits of the pixel value of the grid image in the image sequence are located after the first bit of the pixel value of the grid image in the image sequence and are adjacent to the first bit, and the first bit is the bit with the highest bit number and the value of 1 in the pixel value of the grid image in the image sequence, or, the N bits of the pixel value of the grid image in the image sequence are located after the highest bit of the pixel value of the grid image in the image sequence and are adjacent to the highest bit of the pixel value of the grid image in the image sequence.

[0308] In addition, under the second relationship, the N bits of the pixel value of the grid image in the image sequence are in the same position as the N bits of the pixel value of the grid image in the processed image sequence, the number of bits of the i-th bit is higher than the number of bits of the i+1-th bit, i and N are both positive integers, i≤N, N<M.

[0309] In the present application, the encoding end performs XOR processing on the pixel values ​​of the grid image, so as to reduce the data amount of the grid image when the pixel values ​​of the grid image contain continuous placeholders, thereby making the encoded code stream occupy less storage space, and the decoding end performs (XOR) processing on the decoded grid image, so as to restore the grid image processed (XOR processed) by the encoding end, and thus obtain the final grid image.

[0310] exist Fig.11 In the method shown, in addition to processing the grid images in the image sequence after decoding the grid images in the image sequence, the residual image of the grid image in the grid image can also be processed after decoding the residual image, which is described in detail below.

[0311] like Fig.13 As shown, after step 1002, step 1003a may be performed.

[0312] 1003a. Process the residual image to obtain a processed residual image.

[0313] When step 1004 is executed after step 1003a, step 1004 is to obtain the grid image in the image sequence according to the processed residual image and the predicted image of the grid image in the image sequence.

[0314] The pixel value of the residual image is represented by M bits, each bit has a value of 0 or 1, and M is an integer greater than 1.

[0315] The processed residual image and the residual image obtained in the above step 1003a can satisfy the third relationship.

[0316] Third relationship: The pixel values ​​of the processed residual image are opposite to the pixel values ​​of the residual image.

[0317] Specifically, under the third relationship, the value of the i-th bit in the N bits of the pixel value of the residual image is opposite to the value of the i-th bit in the N bits of the pixel value of the processed residual image, and the N bits of the pixel value of the residual image are located in the same position as the N bits of the pixel value of the processed residual image. The N bits of the pixel value of the residual image are located after the first bit of the pixel value of the residual image and are adjacent to the first bit, the first bit is the bit with the highest bit value of 1 in the pixel value of the residual image, i and N are both positive integers, i≤N, N<M.

[0318] In the present application, the encoding end performs inversion processing on the pixel values ​​of the residual image, so that when the pixel values ​​of the residual image contain continuous placeholders, the data amount of the residual image can be reduced, thereby making the bit stream generated by the encoding occupy less storage space, and the decoding end performs (inversion) processing on the decoded residual image, so as to restore the residual image processed (inverted) by the encoding end, and then obtain the final image block according to the processed residual image.

[0319] In the above step 1003a, in addition to performing an inversion process on the pixel values ​​of the residual image, an XOR process may also be performed on the pixel values ​​of the residual image.

[0320] Therefore, the processed residual image and the residual image obtained in the above step 1003a may satisfy not only the third relationship but also the fourth relationship.

[0321] Fourth relationship: The pixel value of the processed residual image is the XOR result of the pixel value.

[0322] Specifically, under the fourth relationship, the value of the first bit among the N bits of the processed residual image is opposite to the value of the first bit among the N bits of the residual image, and the value of the i+1th bit among the N bits of the processed residual image is the result of XOR processing of the value of the i+1th bit among the N bits of the residual image and the i-th bit among the N bits of the processed residual image. The N bits of the pixel value of the residual image are located after the first bit of the pixel value of the residual image and are adjacent to the first bit, and the first bit is the bit with the highest bit value of 1 in the pixel value of the residual image, or the N bits of the pixel value of the residual image are located after the highest bit of the pixel value of the residual image and are adjacent to the highest bit of the pixel value of the residual image.

[0323] In addition, under the fourth relationship, the N bits of the pixel value of the residual image are located at the same position as the N bits of the pixel value of the processed residual image, the number of bits of the i-th bit is higher than the number of bits of the i+1-th bit, i and N are both positive integers, i≤N, N<M;

[0324] In the present application, the encoder performs XOR processing on the pixel values ​​of the residual image, so that when the pixel values ​​of the residual image contain continuous placeholders, the data amount of the residual image can be reduced, thereby making the bitstream generated by the encoding occupy less storage space, and the decoder performs (XOR) processing on the decoded residual image, so as to restore the residual image processed (XOR processed) by the encoder, and then obtain the final image block according to the processed residual image.

[0325] In the present application, the encoder can use either lossless coding or lossy coding when encoding the grid images in the image sequence. Correspondingly, the decoder should use a method matching that of the encoder to perform decoding.

[0326] Optionally, as an embodiment, in the above step 1002, obtaining a residual image of the grid image in the image sequence according to the bitstream includes: performing inverse transformation, inverse quantization and entropy decoding processing on the bitstream to obtain the residual image.

[0327] The above decoding process corresponds to lossy encoding of the image, which can make the encoded code stream occupy as little storage space as possible.

[0328] Optionally, as an embodiment, in the above step 1002, obtaining a residual image of the grid image in the image sequence according to the bitstream includes: performing entropy decoding processing on the bitstream to obtain the residual image.

[0329] The above decoding method of only performing entropy decoding corresponds to lossless coding, which can avoid image distortion as much as possible and ensure the final display effect of the image.

[0330] Optionally, as an embodiment, the above code stream is obtained by encoding the positioning layer.

[0331] Optionally, as an embodiment, the pixel value of the above positioning layer includes rasterized elevation data.

[0332] The pixel value of the positioning layer contains M bits, each bit representing the occupied position of each grid. Since the pixel value of the positioning layer contains the rasterized elevation data, and the probability of the character "11" appearing in the high position of the elevation data is relatively high, the pixel value of the positioning layer containing the elevation data is processed to reduce the data volume of the pixel value of the positioning layer, thereby reducing the storage space occupied by the final code stream.

[0333] Optionally, in the above first to fourth relationships, the value of M can be any one of 8, 10 and 12.

[0334] Combination of the above Figures 11 to 13 The image decoding method of the embodiment of the present application is described in detail. Fig.14 The image encoding method of the embodiment of the present application is described in detail from the perspective of the encoding end.

[0335] Fig.14 It is a schematic flowchart of the image encoding method of an embodiment of the present application. Fig.14 The method shown may be executed by an encoding device or an encoder. Fig.14 The method shown includes steps 2001 to 2003, and steps 2001 to 2003 are described in detail below.

[0336] 2001. Obtain image sequence.

[0337] 2002. Inter-frame prediction is performed on the grid images in the image sequence to obtain a residual image.

[0338] Optionally, the above-mentioned inter-frame prediction of the grid images in the image sequence to obtain the residual image includes: performing inter-frame prediction on the grid images in the image sequence to obtain a predicted image; and obtaining a residual image of the grid image based on the grid images in the image sequence and the predicted image.

[0339] After the predicted image is obtained, the grid image and the residual image of the grid image may be subtracted to obtain the residual image of the grid image.

[0340] In addition, the above-mentioned image sequence includes multiple grid images, and the geographical locations corresponding to the multiple grid images are close or the image contents of the multiple grid images are similar. Specifically, the geographical locations corresponding to the above-mentioned multiple grid images are close or the image contents of the multiple grid images are similar, which means that the above-mentioned multiple grid images meet the first condition.

[0341] The first condition mentioned above can be expressed in many different specific forms.

[0342] Optionally, the first condition may be that a distance between geographical locations corresponding to at least two grid images in the plurality of grid images in the image sequence is less than a preset distance.

[0343] The above-mentioned preset distance may also be a distance threshold, and the size of the preset distance may be flexibly set according to actual needs. For example, the above-mentioned preset distance may be set to 3m, and when the distance between the corresponding geographical locations of two grid images is less than 3m, the two grid images meet the first condition. The 3m here is only an example, and the preset distance may also be set to 5m and 6m, etc. In short, any suitable preset distance may be set according to needs.

[0344] In addition, the first condition may also be that a distance between geographical locations corresponding to at least two grid images among the plurality of grid images is less than or equal to a preset distance.

[0345] Since images with similar geographical locations are generally highly similar, when an image sequence includes at least two grid images with corresponding geographical locations being adjacent, performing inter-frame prediction on the image sequence can reduce the amount of data of the residual image obtained subsequently.

[0346] Optionally, the first condition may be that the similarity of image contents of at least two grid images in the image sequence is greater than or equal to a preset similarity.

[0347] The above-mentioned preset similarity can also be called a similarity threshold, and the size of the preset similarity can be flexibly set according to actual needs. For example, the above-mentioned preset similarity can be set to 50%. When the similarity of the image content of the two grid images is greater than or equal to 50%, the two grid images meet the first condition. The 50% here is only an example, and the preset similarity can also be set to 55%, 60%, 65%, etc. In short, any suitable size of preset similarity can be set according to needs.

[0348] In addition, the first condition may also be that the similarity between the image contents of at least two grid images among the plurality of grid images is greater than a preset similarity.

[0349] When the image sequence includes at least two grid images with similar image contents, the data volume of the residual image obtained subsequently can be reduced by performing inter-frame prediction on the image sequence.

[0350] Optionally, the first condition may be that image contents of at least two grid images among the plurality of grid images correspond to the same geographical area.

[0351] Since the similarity of images in the same geographical area is generally high, when the image sequence includes at least two grid images located in the same geographical area, the data volume of the residual image obtained subsequently can be smaller by performing inter-frame prediction on the image sequence.

[0352] The size of the geographic area may be a preset size, and the size of the geographic area may be set according to actual needs. For example, the size of the geographic area may be a 2m×2m area. It should be understood that the size of the geographic area of ​​2m×2m here is only an example, and the size of the geographic area may also be set to a 1m×1m area, a 3m×32m area, etc. In short, any suitable size of the geographic area may be set according to needs.

[0353] For another example, the size of the above-mentioned geographical area is an area of ​​2m×2m. When the two grid images are both located in the area of ​​2m×2m, the two grid images meet the first condition.

[0354] Furthermore, the at least two grid images mentioned above may be grid images of adjacent frames.

[0355] That is to say, at least two grid images satisfying the first condition are adjacent frames to each other. For example, grid image 1, grid image 2 and grid image 3 in the above image sequence satisfy the above first condition. Then, grid image 1, grid image 2 and grid image 3 may be the i-th frame, i+1-th frame and i+2-th frame in the image sequence, respectively. Where i is a positive integer.

[0356] 2003. Encode the residual image to obtain a coded bit stream.

[0357] In the present application, since the image sequence contains grid images with similar geographical locations or image contents, when inter-frame prediction is performed on the image sequence, the amount of data of the obtained residual image can be smaller, and thus the code stream obtained by encoding the image sequence finally occupies less storage space, which is convenient for storage or transmission.

[0358] exist Fig.14In the method shown, the prediction image of the grid image may be composed of the prediction blocks of each image block in the grid image, and the residual image of the grid image may be composed of the residual blocks of each image block in the grid image. When encoding is performed at the encoding end, the residual image may be divided into image blocks, and then the prediction block of each image block is obtained, and the residual block of each image block is obtained according to each image block and the residual block of each image block, thereby obtaining the residual image of the grid image.

[0359] In an embodiment of the present application, the encoding end may also perform inversion processing or XOR processing on the pixel values ​​of the grid images in the image sequence before or during encoding the grid images in the image sequence to further reduce the storage space occupied by the code stream generated by the encoding.

[0360] Specifically, in Fig.14 In the method shown, before encoding the grid images in the image sequence, the grid images in the image sequence may be processed first, and then the processed grid images may be encoded.

[0361] like Fig.15 As shown, before executing step 2002, step 2002a is executed first.

[0362] 2002a. Process the grid image in the image sequence to obtain a processed grid image.

[0363] After step 2002a, when step 2002 is executed again, step 2002 is to perform inter-frame prediction based on the processed grid image to obtain a residual image.

[0364] The pixel value of the grid image in the above image sequence is represented by M bits, each bit has a value of 0 or 1, and M is an integer greater than 1.

[0365] The processed grid image and the grid image obtained in the above step 2002a can satisfy the fifth relationship.

[0366] Fifth relationship: The pixel values ​​of the processed grid image are opposite to the pixel values ​​of the grid image.

[0367] Specifically, under the fifth relationship, the value of the i-th bit among the N bits of the pixel value of the grid image in the image sequence is opposite to the value of the i-th bit among the N bits of the pixel value of the grid image in the processed image sequence, and the N bits of the pixel value of the grid image in the image sequence and the N bits of the pixel value of the grid image in the processed image sequence are in the same position.

[0368] In addition, under the fifth relationship, the N bits of the pixel value of the grid image in the image sequence are located after and adjacent to the first bit of the pixel value of the grid image in the image sequence, the first bit is the bit with a value of 1 and the highest number of bits in the pixel value of the grid image in the image sequence, i and N are both positive integers, i≤N, N<M.

[0369] In the present application, the encoding end can reduce the data volume of the grid image by inverting the pixel values ​​of the grid image when the pixel values ​​of the grid image contain continuous placeholders, thereby making the bit stream generated by the encoding occupy less storage space.

[0370] Furthermore, the corresponding coding performance is different when N takes different values. When N takes a certain specific value, the corresponding coding performance is optimal and the number of bits reduced is the largest.

[0371] like Fig.16 When the pixel values ​​of the grid image correspond to 8 bits in total, the number of bits can be reduced by a certain amount when the N bits in the pixel values ​​of the grid image are inverted. When N=3, the degree of bit reduction is the largest, and the corresponding encoding performance is also optimal.

[0372] In the actual encoding process, the value of N can be set according to the test results or experience. For example, when the pixel values ​​of the grid image correspond to 8 bits in total, N=3 or N=4 can be selected. This can achieve better encoding performance and reduce the amount of data of the grid image as much as possible.

[0373] In the present application, in addition to performing an inversion process on the pixel values ​​of the grid images in the image sequence, an XOR process may also be performed on the pixel values ​​of the grid images in the image sequence.

[0374] Therefore, the processed grid image and the grid image obtained in the above step 2002a can satisfy not only the fifth relationship but also the sixth relationship.

[0375] Sixth relationship: The pixel value of the processed grid image is the XOR result of the pixel values.

[0376] Specifically, under the sixth relationship, the value of the first bit among the N bits of the grid image in the processed image sequence is opposite to the value of the first bit among the N bits of the grid image in the image sequence, and the value of the i+1th bit among the N bits of the grid image in the processed image sequence is the result of XOR processing of the value of the i+1th bit among the N bits of the grid image in the image sequence and the i-th bit among the N bits of the grid image in the processed image sequence.

[0377] In addition, under the sixth relationship, the N bits of the pixel value of the grid image in the image sequence are located after and adjacent to the first bit of the pixel value of the grid image in the image sequence, and the first bit is the bit with a value of 1 and the highest bit number in the pixel value of the grid image in the image sequence, or the N bits of the pixel value of the grid image in the image sequence are located after and adjacent to the highest bit number of the pixel value of the grid image in the image sequence. The N bits of the pixel value of the grid image in the image sequence are located in the same position as the N bits of the pixel value of the grid image in the processed image sequence, the number of bits of the i-th bit is higher than the number of bits of the i+1-th bit, i and N are both positive integers, i≤N, N<M.

[0378] In the present application, the encoding end can reduce the data volume of the grid image by performing XOR processing on the pixel values ​​of the grid image when the pixel values ​​of the grid image contain continuous placeholders, thereby making the bit stream generated by the encoding occupy less storage space.

[0379] Furthermore, the corresponding coding performance is different under different values ​​of N. When the value of N is larger, the corresponding coding performance is better and more bits are reduced.

[0380] like Fig.17 When the pixel values ​​of the grid image correspond to 8 bits in total, performing XOR processing on N bits in the pixel values ​​of the grid image can reduce the number of bits to a certain extent. When N=6, the degree of bit reduction is the largest, and the corresponding coding performance is also optimal.

[0381] In the actual encoding process, the value of N can be set according to the test results or experience. For example, when the pixel values ​​of the grid image correspond to 8 bits in total, a larger value can be set for N, such as setting N to 5 or 6. This can achieve better encoding performance and reduce the amount of data of the grid image as much as possible.

[0382] In the image encoding method of the present application, in addition to performing an inversion process or an XOR process on the grid images in the image sequence before encoding the grid images in the image sequence, the residual images may also be inverted or XORed after obtaining the residual images of the grid images in the image sequence. This is described in detail below.

[0383] like Fig.18 As shown, step 2003 may specifically include step 2003a and step 2003b.

[0384] 2003a. Process the residual image to obtain a processed residual image.

[0385] 2003b. Encode the processed residual image to obtain a coded bit stream.

[0386] The pixel value of the residual image is represented by M bits, each bit has a value of 0 or 1, and M is an integer greater than 1.

[0387] The processed residual image and the residual image obtained in the above step 1003a can satisfy the seventh relationship.

[0388] Seventh relationship: The pixel values ​​of the processed residual image are opposite to the pixel values ​​of the residual image.

[0389] Specifically, under the seventh relationship, the value of the i-th bit of the N bits of the pixel value of the residual image is opposite to the value of the i-th bit of the N bits of the pixel value of the processed residual image, and the N bits of the pixel value of the residual image are located at the same position as the N bits of the pixel value of the processed residual image. The N bits of the pixel value of the residual image are located after the first bit of the pixel value of the residual image and are adjacent to the first bit, the first bit is the bit with a value of 1 and the highest bit number in the pixel value of the residual image, i and N are both positive integers, i≤N, N<M.

[0390] In an embodiment of the present application, the encoding end can reduce the amount of data of the residual image by performing XOR processing on the pixel values ​​of the residual image when the pixel values ​​of the residual image contain continuous placeholders, thereby making the bitstream generated by the encoding occupy less storage space.

[0391] In the above step 2003a, in addition to performing the inversion process on the pixel values ​​of the residual image, the pixel values ​​of the residual image may also be subjected to an XOR process.

[0392] Therefore, the processed residual image and the residual image obtained in the above step 2003a can satisfy not only the seventh relationship but also the eighth relationship.

[0393] Eighth relationship: The pixel value of the processed residual image is the XOR result of the pixel value.

[0394] Specifically, under the eighth relationship, the value of the i-th bit among the N bits of the pixel value of the residual image is opposite to the value of the i-th bit among the N bits of the pixel value of the processed residual image, and the value of the i+1-th bit among the N bits of the processed residual image is the result of XOR processing of the value of the i+1-th bit among the N bits of the residual image and the i-th bit among the N bits of the processed residual image.

[0395] In addition, under the eighth relationship, the N bits of the pixel value of the residual image are located after the first bit of the pixel value of the residual image and are adjacent to the first bit, and the first bit is the bit with a value of 1 and the highest bit number in the pixel value of the residual image, or the N bits of the pixel value of the residual image are located after the highest bit of the pixel value of the residual image and are adjacent to the highest bit of the pixel value of the residual image. The N bits of the pixel value of the residual image are located in the same position as the N bits of the pixel value of the processed residual image, the number of bits of the i-th bit is higher than the number of bits of the i+1-th bit, i and N are both positive integers, i≤N, N<M.

[0396] In an embodiment of the present application, the encoding end can reduce the amount of data of the residual image by performing XOR processing on the pixel values ​​of the residual image when the pixel values ​​of the residual image contain continuous placeholders, thereby making the bitstream generated by the encoding occupy less storage space.

[0397] In the image encoding method of the embodiment of the present application, both lossless encoding and lossy encoding can be used.

[0398] Optionally, as an embodiment, encoding the residual image to obtain a coded bitstream in the above step 2003 includes: transforming, quantizing and entropy coding the residual image to obtain a coded bitstream.

[0399] By performing lossy coding, the encoded bit stream can occupy as little storage space as possible.

[0400] Optionally, as an embodiment, encoding the residual image to obtain a coded bitstream in the above step 2003 includes: performing entropy coding processing on the residual image to obtain a coded bitstream.

[0401] Through lossless coding, image distortion can be avoided as much as possible to ensure the final display effect of the image.

[0402] Optionally, as an embodiment, the grid image in the above image sequence comes from a positioning layer.

[0403] Optionally, as an embodiment, the pixel value of the above positioning layer includes rasterized elevation data.

[0404] The pixel value of the positioning layer contains M bits, each bit representing the occupied position of each grid. Since the pixel value of the positioning layer contains the rasterized elevation data, and the probability of the character "11" appearing in the high position of the elevation data is relatively high, the pixel value of the positioning layer containing the elevation data is processed to reduce the data volume of the pixel value of the positioning layer, thereby reducing the storage space occupied by the final code stream.

[0405] Optionally, in the fifth to eighth relationships above, the value of M may be any one of 8, 10 and 12.

[0406] In the embodiment of the present application, the grid images in the image sequence may include I frame images and P frame images. Fig.19 As shown, the image sequence includes 1 I frame and 3 P frames, and the P frames can be predicted by using inter-frame prediction. Fig.19 The image sequence shown is for illustration only, and the embodiment of the present application does not limit the number of grid images contained in the image sequence.

[0407] In addition, the image sequence in the embodiment of the present application may also include B frames.

[0408] In an embodiment of the present application, the grid image in the image sequence may come from a positioning layer, and the positioning layer may be 2D plane data obtained by mapping 3D point cloud data.

[0409] Generally speaking, due to the huge amount of data, 3D point cloud data often needs to be converted into 2D plane data as a positioning layer. Fig. 20 As shown, 3D point cloud data contains data such as three-dimensional coordinates (x, y, z) and reflectivity (r). When converted into 2D plane data, the (x, y) of the 3D point cloud data can be directly used as the (x, y) coordinates of the 2D plane data (x and y are equivalent to the position of the pixel in the picture), and the height z in the 3D point cloud data is mapped to the R channel of the 2D plane data, and the reflectivity r is mapped to the G channel. In this way, the conversion of 3D point cloud data to 2D plane data is realized.

[0410] The effect of the image encoding method of the embodiment of the present application is described below in conjunction with specific test results.

[0411] As shown in Table 1, sequence 1 and sequence 2 are two image sequences, both of which contain images in portable network graphics (PNG) format, wherein sequence 1 contains 4 grid images (1a.png, 1b.png, 1c.png, 1d.png), and sequence 2 also includes 4 grid images (2a.png, 2b.png, 2c.png, 2d.png).

[0412] When encoding sequence 1 and sequence 2, compared with high efficiency video coding-screen content coding (HEVC-SCC) single compression (traditional solution), when the image sequence is encoded using HEVC-SCC combined compression (equivalent to the encoding method of the embodiment of the present application), the data amount of the obtained code stream is smaller.

[0413] For example, for sequence 1, the amount of data after HEVC-SCC combined compression is reduced by 9.9% relative to the amount of data of the original image sequence, and the amount of data after HEVC-SCC alone is compressed is reduced by 9.5% relative to the amount of data of the original image sequence. Compared with the method of using HEVC-SCC combined compression alone, the amount of data of the code stream obtained by using HEVC-SCC combined compression + data preprocessing (equivalent to the XOR processing or inversion processing of the grid image in the image encoding method of the embodiment of the present application) is smaller than that of the method of using HEVC-SCC combined compression alone. For example, for sequence 1, the amount of data of the code stream obtained by using HEVC-SCC combined compression alone is reduced by 9.9% relative to the amount of data of the original image sequence, while the amount of data of the code stream obtained by using HEVC-SCC combined compression + data preprocessing is reduced by 10.3% relative to the amount of data of the original image sequence.

[0414] Table 1

[0415]

[0416] The above describes in detail the image decoding method and encoding method of the embodiment of the present application in conjunction with the accompanying drawings. Figure 21 to Figure 24 The image decoding device and the image encoding device of the embodiment of the present application are introduced. It should be understood that: Figure 21 to Figure 24 The image decoding device in the embodiment of the present application can perform the image decoding method. Figure 21 to Figure 24 The image encoding device in the embodiment of the present application can perform the image encoding method of the embodiment of the present application. In order to avoid unnecessary repetition, repeated descriptions are appropriately omitted when introducing the image decoding device and the image encoding device of the embodiment of the present application.

[0417] Fig.21 It is a schematic block diagram of an image decoding device according to an embodiment of the present application.

[0418] Fig.21 The image decoding device 10000 shown in the figure includes an acquisition unit 10001 and a processing unit 10002. The image decoding device 10000 can execute the image decoding method of the embodiment of the present application. Specifically, the image decoding device 10000 can execute Figures 11 to 13 The image decoding method shown.

[0419] Fig. 22 It is a schematic block diagram of an image encoding device according to an embodiment of the present application.

[0420] Fig. 22 The image encoding device 11000 shown in the figure includes an acquisition unit 11001 and a processing unit 11002. The image encoding device 11000 can execute the image encoding method of the embodiment of the present application. Specifically, the image encoding device 11000 can execute Fig.14 , Fig.15 and Fig.18 The image encoding method shown.

[0421] Fig.23 It is a schematic diagram of the hardware structure of the image decoding device provided in an embodiment of the present application.

[0422] Fig.23 The image decoding device 12000 shown (the image decoding device 12000 may be a computer device) includes a memory 12001, a memory 12002, a communication interface 12003 and a bus 12004. The memory 12001, the memory 12002 and the communication interface 12003 are connected to each other through the bus 12004.

[0423] The memory 12001 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 12001 may store a program. When the program stored in the memory 12001 is executed by the memory 12002, the memory 12002 is used to execute each step of the image decoding method of the embodiment of the present application.

[0424] The memory 12002 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the image decoding method of the method embodiment of the present application.

[0425] The memory 12002 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the image decoding method of the present application may be completed by an integrated logic circuit of hardware in the memory 12002 or by instructions in the form of software.

[0426] The memory 12002 may also be a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), an off-the-shelf field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The disclosed methods, steps, and logic block diagrams in the embodiments of the present application may be implemented or executed.

[0427] The general processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 12001, and the processor 12002 reads the information in the memory 12001, and completes the functions required to be performed by the units included in the image decoding device in combination with its hardware, or executes the image decoding method of the embodiment of the method of the present application.

[0428] The communication interface 12003 uses a transceiver such as but not limited to a transceiver to implement communication between the image decoding device 12000 and other devices or communication networks. For example, the information of the neural network to be constructed and the training data required in the process of constructing the neural network can be obtained through the communication interface 12003.

[0429] The bus 12004 may include a path for transmitting information between various components of the image decoding device 12000 (for example, the memory 12001, the storage 12002, and the communication interface 12003).

[0430] The acquisition unit 10001 and the processing unit 10002 in the above-mentioned image decoding device 10000 are equivalent to the processor 12002 in the image decoding device 12000.

[0431] Fig.24 It is a schematic diagram of the hardware structure of the image encoding device provided in an embodiment of the present application. Fig.24 The image encoding device 13000 (the image decoding device 13000 may be a computer device) shown includes a memory 13001, a processor 13002, a communication interface 13003 and a bus 13004. The memory 13001, the processor 13002 and the communication interface 13003 are connected to each other through the bus 13004.

[0432] The above definitions and explanations of the various modules in the image decoding device 12000 are also applicable to the image encoding device 13000 and will not be described in detail here.

[0433] The above-mentioned memory 13001 can be used to store programs, and the processor 13002 is used to execute the programs stored in the memory 13001. When the program stored in the memory 13001 is executed, the processor 13002 is used to execute the various steps of the image encoding method of the embodiment of the present application.

[0434] In addition, when the image encoding device 13000 encodes an image, the image can be obtained through the communication interface, and then the obtained image can be encoded to obtain encoded data, and the encoded data can be transmitted to the video decoding device through the communication interface 13003.

[0435] When the image encoding device 13000 decodes an image, it can obtain a video image through a communication interface, and then decode the obtained image to obtain an image to be displayed.

[0436] The acquisition unit 11001 and the processing unit 11002 in the above-mentioned image encoding device 11000 are equivalent to the processor 13002 in the image encoding device 13000.

[0437] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0438] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0439] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0440] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0441] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0442] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0443] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image decoding method, characterized in that: include: Obtaining a bit stream, where the bit stream is obtained by encoding the residual image; Acquire the residual image of the grid image in an image sequence according to the code stream, the image sequence comprising a plurality of grid images, and the plurality of grid images satisfy a first condition; Performing inter-frame prediction on the grid images in the image sequence to obtain predicted images of the grid images in the image sequence; Obtaining a grid image in the image sequence according to a residual image of the grid image in the image sequence and a predicted image of the grid image in the image sequence; The first condition includes: the distance between the geographical locations corresponding to at least two of the multiple grid images is less than a preset distance; or the image contents of at least two of the multiple grid images correspond to the same geographical area.

2. The method according to claim 1, characterized in that At least two grid images among the plurality of grid images are grid images of adjacent frames.

3. The method according to any one of claims 1 to 2, characterized in that: The pixel values ​​of the grid images in the image sequence are represented by M bits, each bit has a value of 0 or 1, and M is an integer greater than 1. The method further includes: Processing the grid images in the image sequence to obtain processed grid images; The value of the i-th bit in the N bits of the pixel value of the grid image in the image sequence is opposite to the value of the i-th bit in the N bits of the pixel value of the grid image in the processed image sequence, and the N bits of the pixel value of the grid image in the image sequence and the N bits of the pixel value of the grid image in the processed image sequence are located in the same position, The N bits of the pixel value of the grid image in the image sequence are located after the first bit of the pixel value of the grid image in the image sequence and are adjacent to the first bit, the first bit is the bit with a value of 1 and the highest number of bits in the pixel value of the grid image in the image sequence, i and N are both positive integers, i≤N, N<M.

4. The method according to any one of claims 1 to 2, characterized in that: The pixel values ​​of the grid images in the image sequence are represented by M bits, each bit has a value of 0 or 1, and M is an integer greater than 1. The method further includes: Processing the grid images in the image sequence to obtain processed grid images; The value of the first bit among the N bits of the grid image in the processed image sequence is opposite to the value of the first bit among the N bits of the grid image in the image sequence, and the value of the i+1th bit among the N bits of the grid image in the processed image sequence is the result of exclusive-OR processing of the value of the i+1th bit among the N bits of the grid image in the image sequence and the i-th bit among the N bits of the grid image in the processed image sequence. The N bits of the pixel value of the grid image in the image sequence are located after the first bit of the pixel value of the grid image in the image sequence and are adjacent to the first bit, and the first bit is the bit with a value of 1 and the highest bit number in the pixel value of the grid image in the image sequence, or the N bits of the pixel value of the grid image in the image sequence are located after the highest bit of the pixel value of the grid image in the image sequence and are adjacent to the highest bit of the pixel value of the grid image in the image sequence, The N bits of the pixel values ​​of the grid image in the image sequence are in the same position as the N bits of the pixel values ​​of the grid image in the processed image sequence, the number of bits of the i-th bit is higher than the number of bits of the i+1-th bit, i and N are both positive integers, i≤N, N<M.

5. The method according to any one of claims 1 to 2, characterized in that: The pixel value of the residual image is represented by M bits, each bit has a value of 0 or 1, and M is an integer greater than 1. The grid image in the image sequence is obtained according to the residual image of the grid image in the image sequence and the predicted image of the grid image in the image sequence, including: Processing the residual image to obtain a processed residual image; Obtaining a grid image in the image sequence according to the processed residual image and a predicted image of the grid image in the image sequence; The value of the i-th bit of the N bits of the pixel value of the residual image is opposite to the value of the i-th bit of the N bits of the pixel value of the processed residual image, and the N bits of the pixel value of the residual image and the N bits of the pixel value of the processed residual image are located in the same position, The N bits of the pixel value of the residual image are located after the first bit of the pixel value of the residual image and are adjacent to the first bit, the first bit is the bit with a value of 1 and the highest bit number in the pixel value of the residual image, i and N are both positive integers, i≤N, N<M.

6. The method according to any one of claims 1 to 2, characterized in that: The pixel value of the residual image is represented by M bits, each bit has a value of 0 or 1, and M is an integer greater than 1. The grid image in the image sequence is obtained according to the residual image of the grid image in the image sequence and the predicted image of the grid image in the image sequence, including: Processing the residual image to obtain a processed residual image; Obtaining a grid image in the image sequence according to the processed residual image and a predicted image of the grid image in the image sequence; The value of the first bit among the N bits of the processed residual image is opposite to the value of the first bit among the N bits of the residual image, and the value of the i+1th bit among the N bits of the processed residual image is the result of XOR processing of the value of the i+1th bit among the N bits of the residual image and the i-th bit among the N bits of the processed residual image. The N bits of the pixel value of the residual image are located after the first bit of the pixel value of the residual image and are adjacent to the first bit, and the first bit is the bit with a value of 1 and the highest bit number in the pixel value of the residual image, or the N bits of the pixel value of the residual image are located after the highest bit of the pixel value of the residual image and are adjacent to the highest bit of the pixel value of the residual image, The N bits of the pixel value of the residual image are in the same position as the N bits of the pixel value of the processed residual image, the number of bits of the i-th bit is higher than the number of bits of the i+1-th bit, i and N are both positive integers, i≤N, N<M.

7. The method according to any one of claims 1 to 2, characterized in that: The step of obtaining a residual image of a grid image in an image sequence according to the code stream includes: The code stream is subjected to inverse transformation, inverse quantization and entropy decoding to obtain the residual image.

8. The method according to any one of claims 1 to 2, characterized in that The step of obtaining a residual image of a grid image in an image sequence according to the code stream includes: Perform entropy decoding on the code stream to obtain the residual image.

9. The method according to any one of claims 1 to 2, characterized in that: The code stream is obtained by encoding the positioning layer.

10. The method according to claim 9, characterized in that The pixel values ​​of the positioning layer include rasterized elevation data.

11. The method according to any one of claims 1 to 2, characterized in that: The value of M can be any one of 8, 10 and 12.

12. An image encoding method, characterized in that: include: Acquire an image sequence, the image sequence comprising a plurality of grid images, the plurality of grid images satisfying a first condition; Performing inter-frame prediction on the grid images in the image sequence to obtain a residual image; Encoding the residual image to obtain an encoded bitstream; The first condition includes: the distance between the geographical locations corresponding to at least two of the multiple grid images is less than a preset distance, or the image contents of at least two of the multiple grid images correspond to the same geographical area.

13. The method according to claim 12, characterized in that At least two grid images among the plurality of grid images are grid images of adjacent frames.

14. The method according to any one of claims 12 to 13, characterized in that The pixel values ​​of the grid images in the image sequence are represented by M bits, each bit has a value of 0 or 1, and M is an integer greater than 1. Before performing inter-frame prediction on the grid images in the image sequence, the method further includes: Processing the grid images in the image sequence to obtain processed grid images; The value of the i-th bit in the N bits of the pixel value of the grid image in the image sequence is opposite to the value of the i-th bit in the N bits of the pixel value of the grid image in the processed image sequence, and the N bits of the pixel value of the grid image in the image sequence and the N bits of the pixel value of the grid image in the processed image sequence are located in the same position, N bits of the pixel value of the grid image in the image sequence are located after the first bit of the pixel value of the grid image in the image sequence and are adjacent to the first bit, the first bit is the bit with a value of 1 and the highest bit number in the pixel value of the grid image in the image sequence, i and N are both positive integers, i≤N, N<M; The performing inter-frame prediction on the grid images in the image sequence comprises: Inter-frame prediction is performed on the processed grid image to obtain the residual image.

15. The method according to any one of claims 12 to 13, characterized in that The pixel values ​​of the grid images in the image sequence are represented by M bits, each bit has a value of 0 or 1, and M is an integer greater than 1. Before performing inter-frame prediction on the grid images in the image sequence, the method further includes: Processing the grid images in the image sequence to obtain processed grid images; The value of the first bit among the N bits of the grid image in the processed image sequence is opposite to the value of the first bit among the N bits of the grid image in the image sequence, and the value of the i+1th bit among the N bits of the grid image in the processed image sequence is the result of exclusive-OR processing of the value of the i+1th bit among the N bits of the grid image in the image sequence and the i-th bit among the N bits of the grid image in the processed image sequence. The N bits of the pixel value of the grid image in the image sequence are located after the first bit of the pixel value of the grid image in the image sequence and are adjacent to the first bit, and the first bit is the bit with a value of 1 and the highest bit number in the pixel value of the grid image in the image sequence, or the N bits of the pixel value of the grid image in the image sequence are located after the highest bit of the pixel value of the grid image in the image sequence and are adjacent to the highest bit of the pixel value of the grid image in the image sequence, The N bits of the pixel values ​​of the grid images in the image sequence are located at the same position as the N bits of the pixel values ​​of the grid images in the processed image sequence, the number of bits of the i-th bit is higher than the number of bits of the i+1-th bit, i and N are both positive integers, i≤N, N<M; The performing inter-frame prediction on the grid images in the image sequence comprises: Inter-frame prediction is performed on the processed grid image to obtain the residual image.

16. The method according to any one of claims 12 to 13, characterized in that: The pixel value of the residual image is represented by M bits, each bit has a value of 0 or 1, and M is an integer greater than 1. The residual image is encoded to obtain an encoded bit stream, including: Processing the residual image to obtain a processed residual image; Encoding the processed residual image to obtain the encoded bitstream; The value of the i-th bit of the N bits of the pixel value of the residual image is opposite to the value of the i-th bit of the N bits of the pixel value of the processed residual image, and the N bits of the pixel value of the residual image and the N bits of the pixel value of the processed residual image are located in the same position, The N bits of the pixel value of the residual image are located after the first bit of the pixel value of the residual image and are adjacent to the first bit, the first bit is the bit with a value of 1 and the highest bit number in the pixel value of the residual image, i and N are both positive integers, i≤N, N<M.

17. The method according to any one of claims 12 to 13, characterized in that: The pixel value of the residual image is represented by M bits, each bit has a value of 0 or 1, and M is an integer greater than 1. The residual image is encoded to obtain an encoded bit stream, including: Processing the residual image to obtain a processed residual image; Encoding the processed residual image to obtain the encoded bitstream; The value of the i-th bit in the N bits of the pixel value of the residual image is opposite to the value of the i-th bit in the N bits of the pixel value of the processed residual image, and the value of the i+1-th bit in the N bits of the processed residual image is the result of XOR processing of the value of the i+1-th bit in the N bits of the residual image and the i-th bit in the N bits of the processed residual image. The N bits of the pixel value of the residual image are located after the first bit of the pixel value of the residual image and are adjacent to the first bit, and the first bit is the bit with a value of 1 and the highest bit number in the pixel value of the residual image, or the N bits of the pixel value of the residual image are located after the highest bit of the pixel value of the residual image and are adjacent to the highest bit of the pixel value of the residual image, The N bits of the pixel value of the residual image are in the same position as the N bits of the pixel value of the processed residual image, the number of bits of the i-th bit is higher than the number of bits of the i+1-th bit, i and N are both positive integers, i≤N, N<M.

18. The method according to any one of claims 12 to 13, characterized in that: The residual image is encoded to obtain an encoded bitstream, including: The residual image is transformed, quantized and entropy encoded to obtain an encoded bit stream.

19. The method according to any one of claims 12 to 13, characterized in that: The residual image is encoded to obtain an encoded bitstream, including: The residual image is entropy encoded to obtain an encoded bit stream.

20. The method according to any one of claims 12 to 13, characterized in that The grid images in the image sequence are from the positioning layer.

21. The method of claim 20, wherein: The pixel values ​​of the positioning layer include rasterized elevation data.

22. The method according to any one of claims 12 to 13, characterized in that The value of M can be any one of 8, 10 and 12.

23. An image decoding device, characterized in that: The image decoding device comprises a module for executing the method according to any one of claims 1-11.

24. An image encoding device, characterized in that: The image encoding device comprises modules for executing the method according to any one of claims 12-22.

25. An image decoding device, characterized in that: include: Memory, used to store programs; A processor, configured to execute the program stored in the memory. When the program stored in the memory is executed by the processor, the processor executes the method according to any one of claims 1 to 11.

26. An image encoding device, characterized in that include: Memory, used to store programs; A processor, configured to execute the program stored in the memory; when the program stored in the memory is executed by the processor, the processor executes the method as described in any one of claims 12 to 22.

27. An electronic device, characterized in that: The electronic device includes the image decoding device according to claim 25 and / or the image encoding device according to claim 26.

28. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that can be executed by a processor. When the computer program is executed by the processor, the processor executes the method according to any one of claims 1 to 11 or 12 to 22.

Images