Layered coding and decoding method and device

Abstract

The invention provides a layered coding and decoding method and device, and belongs to the technical field of video compression, and the method comprises the steps: carrying out the prediction coding of a basic layer on a plurality of first CUs obtained by dividing a first image, obtaining a basic layer bit stream, and carrying out the inter-frame prediction of the first CU, if a reference pixel of the first CU is located in a second region of a reference image, carrying out the inter-frame prediction of the first CU; the reference pixel of the first CU adopts a preset pixel value; if the reference motion information of the first CU comes from the second area, the reference motion information is invalid; performing prediction coding of an enhancement layer on a plurality of second CUs obtained by dividing the second image to obtain an enhancement layer bit stream, when the second CUs are located in the first region, a prediction mode of the second CUs comprises inter-layer prediction, and reference blocks of the second CUs are reconstructed blocks in the first reconstructed image, the positions of which are the same as those of the second CUs; and when the second CU is located in the second region, the prediction mode of the second CU comprises inter-frame prediction and intra-frame prediction. According to the invention, the coding and decoding cost is further reduced on the premise of ensuring the user information security.

Description

Technical Field

[0001] The present application relates to the field of video compression technology, and in particular to a layered coding and decoding method and device. Background Art

[0002] With the rapid development of the Internet and multimedia technology, as well as the continuous decline in the hardware cost and implementation cost of video surveillance, video surveillance technology has been widely used; for example, video surveillance technology is used to implement remote video surveillance of target areas to assist in public security management, accident warning, etc.

[0003] However, the collected video images may carry user information (such as faces, license plate numbers, etc.). If the video images are leaked, the user information will also be leaked, which may cause losses to the user; therefore, video privacy protection technology came into being.

[0004] In the prior art, the encoding end inputs the privacy area image and the non-privacy area image of the video image into two encoders for encoding, and obtains a bit stream of the privacy area image and a bit stream of the non-privacy area image; then, the bit stream of the non-privacy area image and the encrypted bit stream of the privacy area image are transmitted; in this way, video privacy protection can be achieved. However, the privacy area image and the non-privacy area image are encoded independently, which reduces the encoding performance to a certain extent; in addition, the system layer needs to synchronize the bit stream of the privacy area image with the bit stream of the non-privacy area image to ensure that the decoding end can obtain a complete image, which will bring computing overhead. Summary of the invention

[0005] The present application provides a layered coding and decoding method and device, which solves the problem of large coding and decoding operation overhead in related technologies, and can further reduce the coding and decoding cost while ensuring user information security.

[0006] In a first aspect, the present application provides a layered coding method, the method comprising: obtaining a first image and a second image, the first image comprising an image of a first region in an original image and excluding an image of a second region in the original image, the second image comprising an image of the second region in the original image, the size of the first image and the size of the second image being the same as the size of the original image, and the image of the second region in the original image comprising user information; dividing the first image to obtain a plurality of first coding units (coding units); unit, CU), any one of the plurality of first CUs is located in the first area or in the second area; base layer prediction coding is performed on the plurality of first CUs to obtain a base layer bit stream, and the prediction mode of the first CU includes: inter-frame prediction and intra-frame prediction; when inter-frame prediction is performed on the first CU, if the reference pixel of the first CU is located in the second area of ​​the reference image, the pixel value of the reference pixel of the first CU adopts a preset pixel value; if the reference motion information of the first CU comes from the second area, the reference motion information is set to be invalid; the second image is divided to obtain a plurality of second CUs, any one of the plurality of second CUs is located in the first area or in the second area; enhancement layer prediction coding is performed on the plurality of second CUs to obtain an enhancement layer bit stream, when the second CU is located in the first area, the prediction mode of the second CU includes: inter-layer prediction, and the reference block of the second CU is a reconstructed block in the first reconstructed image with the same position as the second CU, and the first reconstructed image is a reconstructed image of the first image; when the second CU is located in the second area, the prediction mode of the second CU includes: inter-frame prediction and intra-frame prediction; wherein the base layer bit stream and the enhancement layer bit stream are interwoven into one bit stream.

[0007] Exemplarily, the reference image is the first reconstructed image of the encoded frame before the current frame, and the preset pixel value may be, for example, a median value or a value transmitted through the image header of the first image, as long as the preset pixel values ​​of the encoder and the decoder are the same.

[0008] The beneficial effect is that the reconstructed pixel values ​​(after loop filtering) and motion information in the first area of ​​the second image stored in the decoded picture buffer (DPB) can be guaranteed to be completely consistent with the first image. And even if the image in the second area of ​​the first image does not exist, the codec can still obtain consistent pixel values. As a result, BL and EL can reuse one DPB, solving the problem of high encoding and decoding costs caused by doubling DPB resources in layered coding, thereby further reducing encoding and decoding costs.

[0009] In one possible implementation, a process of performing prediction encoding of a base layer on multiple first CUs to obtain a base layer bit stream includes: performing inter-frame prediction or intra-frame prediction on multiple first CUs to obtain a predicted image of the first image; encoding the residual between the first image and the predicted image of the first image to obtain a base layer bit stream.

[0010] Optionally, the reconstructed pixels in the second area may be filled in a manner agreed upon by the codec, such as being filled with a certain color value, and the color value may be transmitted in the bitstream.

[0011] The beneficial effect is that by further agreeing on the reconstructed pixel filling value in the second area of ​​the BL layer at the codec end, it is only necessary to encode the content of the EL layer once, and the image in the first area can be obtained by encoding the BL layer, and the image in the second area can be obtained by encoding the EL layer. The purpose of one-time encoding and one-time decoding is achieved, and the implementation cost is further reduced.

[0012] In one possible implementation, the process of performing inter-frame prediction or intra-frame prediction on multiple first CUs to obtain a predicted image of the first image includes: performing inter-frame prediction or intra-frame prediction on a base layer on each first target CU in the multiple first CUs to obtain a prediction block of each first target CU, where the first target CU is a first CU located in the first area; wherein the predicted image of the first image includes the prediction blocks of all first target CUs.

[0013] In a possible implementation, the method further includes: generating a reconstruction block of each first CU in the plurality of first CUs to obtain a first reconstructed image; wherein the first reconstructed image is a reference image of the first image of other frames and / or a reference image of the second image, and the reconstructed boundary pixels between the first region and the second region in the first reconstructed image are reconstructed pixels that have not been loop filtered. Optionally, the reconstructed pixels within the second region in the first image are reconstructed pixels that have not been loop filtered.

[0014] The beneficial effect thereof is that it can prevent the preset value filled in the second area in the first reconstructed image from contaminating the pixel values ​​in the second area.

[0015] In a possible implementation manner, the reconstructed pixels of the second area in the first reconstructed image are preset values.

[0016] In a possible implementation manner, when loop filtering is performed on the reconstructed pixels in the first area of ​​the first reconstructed image, the referenced reconstructed pixels do not include the reconstructed pixels in the second area.

[0017] In one possible implementation, when intra-frame prediction is performed on a first target CU among multiple first CUs, the reference pixels of the first target CU do not include reconstructed pixels within the second region, and the first target CU is the first CU located within the first region; when inter-frame prediction is performed on the first target CU, the reference motion information of the first target CU does not include motion information from the second region.

[0018] In one possible implementation, the process of performing prediction encoding of the enhancement layer on multiple second CUs to obtain an enhancement layer bit stream includes: performing inter-frame prediction, intra-frame prediction or inter-layer prediction on the multiple second CUs to obtain a predicted image of the second image; encoding the residual between the second image and the predicted image of the second image to obtain an enhancement layer bit stream.

[0019] In one possible implementation, the process of performing inter-frame prediction, intra-frame prediction or inter-layer prediction on multiple second CUs to obtain a predicted image of the second image includes: performing inter-frame prediction or intra-frame prediction of an enhancement layer on each second target CU in the multiple second CUs to obtain a prediction block of each second target CU, where the second target CU is a second CU located in the second area; wherein the predicted image of the second image includes the prediction blocks of all second target CUs.

[0020] In a possible implementation, the method also includes: generating a reconstructed block of each second CU in multiple second CUs to obtain a second reconstructed image; wherein the second reconstructed image is a reconstructed image of the second image, the second reconstructed image is a reference image of the second images of other frames, and the pixel value of the reconstructed block of the third target CU in the multiple second CUs is equal to the pixel value of the reconstructed block in the first reconstructed image with the same position as the third target CU; the third target CU is a second CU located in the first area, the image of the second area in the second reconstructed image is an image that has undergone loop filtering, and the image of the first area in the second reconstructed image is an image that has not undergone loop filtering.

[0021] In a possible implementation manner, a residual difference between the prediction block of the third target CU and the first CU is zero.

[0022] The beneficial effect is that the number of bits required to encode the residual between the second CU and the second CU prediction block is small, which can reduce the coding rate overhead of the enhancement layer, thereby reducing the overall coding rate overhead.

[0023] In a possible implementation manner, the second image further includes the image of the first area in the original image; or the second image does not include the image of the first area in the original image.

[0024] In a possible implementation, the method further includes: acquiring position indication information, where the position indication information is used to indicate the position of the second area; and adding the encoded position indication information to the base layer bit stream and / or the enhancement layer bit stream.

[0025] Exemplarily, when the second region is a rectangle, the position indication information may include the coordinates of the upper left corner of the second region and the size of the second region. Exemplarily, the position indication information may be an image block identification map.

[0026] In a possible implementation, the first area in the original image is the area in the original image excluding the second area; the second image is generated by replacing the pixel values ​​of the pixel points in the first area in the original image; the first image is generated by replacing the pixel values ​​of the pixel points in the second area in the original image; wherein the replacement method includes at least one of the following: solid color filling, mosaic or Gaussian blur.

[0027] In one possible implementation, the base layer bitstream and the enhancement layer bitstream include a reference frame identifier and a prediction mode.

[0028] In a second aspect, the present application provides a layered decoding method, which includes: receiving a bit stream, the bit stream includes a base layer bit stream and an enhancement layer bit stream, the base layer bit stream includes multiple first coding blocks, the enhancement layer bit stream includes multiple second coding blocks, the first coding block is a coding block of a first CU, and the second coding block is a coding block of a second CU; performing base layer prediction decoding on the multiple first coding blocks to obtain a first reconstructed image, wherein the first reconstructed image is a reconstructed image of the first image, the first image includes an image of a first area in the original image and does not include an image of a second area in the original image, the image of the second area in the original image includes user information, and the prediction mode of the first coding block includes: inter-frame prediction and intra-frame prediction; when inter-frame prediction is performed on the first coding block, if the reference pixel of the bit stream of the first coding block is located in the second area of ​​the reference image, the pixel value of the reference pixel of the first coding block adopts Preset pixel value; if the reference motion information of the first coding block comes from the second area, the reference motion information is set to invalid; perform prediction decoding of the enhancement layer on multiple second coding blocks to obtain a second reconstructed image, wherein the second reconstructed image is a reconstructed image of the second image, the second image includes an image of the second area in the original image, the size of the first image and the size of the second image are the same as the size of the original image, when the second CU corresponding to the second coding block is located in the first area, the prediction mode of the second coding block includes: inter-layer prediction, and the reference block of the second coding block is a reconstructed block in the first reconstructed image with the same position as the second CU corresponding to the second coding block; when the second CU corresponding to the second coding block is located in the second area, the prediction mode of the second coding block includes: inter-frame prediction and intra-frame prediction; wherein the first reconstructed image and the second reconstructed image are used to determine the reconstructed image of the original image.

[0029] It should be noted that because the first reconstructed image and the second reconstructed image share a DPB, and the reconstructed pixels of the second CU located in the first area in the second image are the same as the reconstructed pixels of the first CU located in the first area in the first image, in one implementation, the decoding of the coding block of the second CU located in the first area in the second image can be skipped to reduce the decoding complexity.

[0030] In one possible implementation, the base layer bitstream includes a reference frame identifier and a prediction mode, and the base layer prediction decoding is performed on multiple first coding blocks to obtain a first reconstructed image. The process includes: decoding multiple first coding blocks to obtain a residual between the first image and a predicted image of the first image; determining the reference image of the base layer bitstream according to the reference frame identifier and the prediction mode in the base layer bitstream; predicting multiple first coding blocks according to the reference image of the base layer bitstream to obtain a predicted image of the first image; adding the predicted image of the first image to the residual decoded from the base layer bitstream to obtain a first reconstructed image.

[0031] In a possible implementation, the multiple first coding blocks are coding blocks of a first target CU in a first image, and the first target CU is a first CU located in a first region; wherein the predicted image of the first image includes prediction blocks of all first target CUs.

[0032] In a possible implementation manner, a residual between the prediction block of the first target CU and the first CU is zero.

[0033] In one possible implementation, the first reconstructed image is a reference image of a base layer bit stream and / or a reference image of an enhanced layer bit stream of a first image of other frames, and the reconstructed boundary pixels between the first area and the second area in the first reconstructed image are reconstructed pixels that have not undergone loop filtering.

[0034] In a possible implementation manner, the reconstructed pixels of the second area in the first reconstructed image are preset values.

[0035] In a possible implementation manner, when loop filtering is performed on the reconstructed pixels in the first area of ​​the first reconstructed image, the referenced reconstructed pixels do not include the reconstructed pixels in the second area.

[0036] In one possible implementation, when intra-frame prediction is performed on the coding block of the first target CU among multiple first coding blocks, the reference pixels of the coding block of the first target CU do not include reconstructed pixels within the second area, and the first target CU is the first CU located within the second area; when inter-frame prediction is performed on the coding block of the first target CU, the reference motion information of the coding block of the first target CU does not include motion information from the second area.

[0037] In one possible implementation, the enhancement layer bitstream includes a reference frame identifier and a prediction mode, and the process of performing enhancement layer prediction decoding on multiple second coding blocks to obtain a second reconstructed image includes: decoding multiple second coding blocks to obtain a residual between the second image and a predicted image of the second image; determining a reference image of the enhancement layer bitstream based on the reference frame identifier and the prediction mode in the enhancement layer bitstream; predicting multiple second coding blocks based on the reference image of the enhancement layer bitstream to obtain a predicted image of the second image; and adding the predicted image of the second image to the residual decoded from the enhancement layer bitstream to obtain a second reconstructed image.

[0038] In a possible implementation, the multiple second coding blocks are coding blocks of a second target CU in the second image, and the second target CU is a second CU located in the second region; wherein the predicted image of the second image includes prediction blocks of all second target CUs.

[0039] In one possible implementation, the second reconstructed image is a reconstructed image of the second image, the second reconstructed image is a reference image of the enhancement layer bit stream of the second image of other frames, the pixel value of the reconstructed block of the coding block of the third target CU in the multiple second coding blocks is equal to the pixel value of the reconstructed block with the same position as the third target CU in the first reconstructed image; the third target CU is the second CU located in the first area, the image of the second area in the second reconstructed image is an image that has undergone loop filtering, and the image of the first area in the second reconstructed image is an image that has not undergone loop filtering.

[0040] In a possible implementation manner, the second image further includes the image of the first area in the original image; or the second image does not include the image of the first area in the original image.

[0041] In a possible implementation, the bitstream further includes encoded position indication information, where the position indication information is used to indicate the position of the second region; the method further includes: determining the second region in the first reconstructed image according to the position indication information.

[0042] In a possible implementation manner, the method further includes: using the second reconstructed image as a reconstructed image of the original image.

[0043] In a third aspect, the present application provides a layered coding device, which includes: an acquisition module, used to acquire a first image and a second image, the first image includes an image of a first area in the original image and does not include an image of a second area in the original image, the second image includes an image of the second area in the original image, the size of the first image and the size of the second image are the same as the size of the original image, and the image of the second area in the original image contains user information; a division module, used to divide the first image to obtain multiple first CUs, any one of the multiple first CUs is located in the first area or in the second area; a prediction coding module, used to perform base layer prediction coding on the multiple first CUs to obtain a base layer bit stream, the prediction mode of the first CU includes: inter-frame prediction and intra-frame prediction; when inter-frame prediction is performed on the first CU, if the reference pixel of the first CU is located in the first area of ​​the reference image, If the reference motion information of the first CU comes from the second area, the pixel value of the reference pixel of the first CU adopts the preset pixel value; if the reference motion information of the first CU comes from the second area, the reference motion information is set to invalid; the division module is also used to divide the second image to obtain multiple second CUs, and any second CU of the multiple second CUs is located in the first area or the second area; the prediction coding module is also used to perform prediction coding of the enhancement layer on the multiple second CUs to obtain the enhancement layer bit stream, when the second CU is located in the first area, the prediction mode of the second CU includes: inter-layer prediction, and the reference block of the second CU is the reconstructed block in the first reconstructed image with the same position as the second CU, and the first reconstructed image is the reconstructed image of the first image; when the second CU is located in the second area, the prediction mode of the second CU includes: inter-frame prediction and intra-frame prediction; wherein the basic layer bit stream and the enhancement layer bit stream are interwoven into one bit stream.

[0044] In a possible implementation, the prediction coding module is specifically used to: perform inter-frame prediction or intra-frame prediction on multiple first CUs to obtain a predicted image of the first image; encode the residual between the first image and the predicted image of the first image to obtain a basic layer bit stream.

[0045] In one possible implementation, the prediction coding module is specifically used to: perform inter-frame prediction or intra-frame prediction of the base layer on each first target CU among multiple first CUs to obtain a prediction block of each first target CU, where the first target CU is the first CU located in the first area; wherein the predicted image of the first image includes the prediction blocks of all first target CUs.

[0046] In one possible implementation, the device also includes: a reconstruction module, used to generate a reconstructed block of each first CU in multiple first CUs to obtain a first reconstructed image; wherein the first reconstructed image is a reference image of the first images of other frames and / or a reference image of the second image, and the reconstructed boundary pixels between the first area and the second area in the first reconstructed image are reconstructed pixels that have not undergone loop filtering.

[0047] In a possible implementation manner, the reconstructed pixels of the second area in the first reconstructed image are preset values.

[0048] In a possible implementation manner, when loop filtering is performed on the reconstructed pixels in the first area of ​​the first reconstructed image, the referenced reconstructed pixels do not include the reconstructed pixels in the second area.

[0049] In one possible implementation, when intra-frame prediction is performed on a first target CU among multiple first CUs, the reference pixels of the first target CU do not include reconstructed pixels within the second region, and the first target CU is the first CU located within the first region; when inter-frame prediction is performed on the first target CU, the reference motion information of the first target CU does not include motion information from the second region.

[0050] In one possible implementation, the prediction coding module is specifically used to: perform inter-frame prediction, intra-frame prediction or inter-layer prediction on multiple second CUs to obtain a predicted image of the second image; encode the residual between the second image and the predicted image of the second image to obtain an enhanced layer bit stream.

[0051] In one possible implementation, the prediction coding module is specifically used to: perform inter-frame prediction or intra-frame prediction of an enhancement layer on each second target CU among multiple second CUs to obtain a prediction block of each second target CU, where the second target CU is a second CU located within the second area; wherein the predicted image of the second image includes the prediction blocks of all second target CUs.

[0052] In a possible implementation, the reconstruction module is also used to generate a reconstructed block of each second CU in multiple second CUs to obtain a second reconstructed image; wherein the second reconstructed image is a reconstructed image of the second image, the second reconstructed image is a reference image of the second images of other frames, and the pixel value of the reconstructed block of the third target CU in the multiple second CUs is equal to the pixel value of the reconstructed block with the same position as the third target CU in the first reconstructed image; the third target CU is a second CU located in the first area, the image of the second area in the second reconstructed image is an image that has undergone loop filtering, and the image of the first area in the second reconstructed image is an image that has not undergone loop filtering.

[0053] In a possible implementation manner, a residual difference between the prediction block of the third target CU and the first CU is zero.

[0054] In a possible implementation manner, the second image further includes the image of the first area in the original image; or the second image does not include the image of the first area in the original image.

[0055] In a possible implementation, the acquisition module is further used to acquire position indication information, where the position indication information is used to indicate the position of the second area; the prediction coding module is further used to add the encoded position indication information to the base layer bit stream and / or the enhancement layer bit stream.

[0056] In a possible implementation, the first area in the original image is the area in the original image excluding the second area; the second image is generated by replacing the pixel values ​​of the pixel points in the first area in the original image; the first image is generated by replacing the pixel values ​​of the pixel points in the second area in the original image; wherein the replacement method includes at least one of the following: solid color filling, mosaic or Gaussian blur.

[0057] In one possible implementation, the base layer bitstream and the enhancement layer bitstream include a reference frame identifier and a prediction mode.

[0058] In a fourth aspect, the present application provides a layered decoding device, which includes: a transceiver module, used to receive a bit stream, the bit stream includes a base layer bit stream and an enhancement layer bit stream, the base layer bit stream includes multiple first coding blocks, the enhancement layer bit stream includes multiple second coding blocks, the first coding block is a coding block of a first CU, and the second coding block is a coding block of a second CU; a prediction decoding module, used to perform base layer prediction decoding on the multiple first coding blocks to obtain a first reconstructed image, wherein the first reconstructed image is a reconstructed image of the first image, the first image includes an image of a first area in the original image and does not include an image of a second area in the original image, the image of the second area in the original image includes user information, and the prediction mode of the first coding block includes: inter-frame prediction and intra-frame prediction; when inter-frame prediction is performed on the first coding block, if the reference pixel of the bit stream of the first coding block is located in the second area of ​​the reference image, the pixel of the reference pixel of the first coding block is The value adopts a preset pixel value; if the reference motion information of the first coding block comes from the second area, the reference motion information is set to invalid; the prediction decoding module is also used to perform prediction decoding of the enhancement layer on multiple second coding blocks to obtain a second reconstructed image, wherein the second reconstructed image is a reconstructed image of the second image, the second image includes an image of the second area in the original image, the size of the first image and the size of the second image are the same as the size of the original image, when the second CU corresponding to the second coding block is located in the first area, the prediction mode of the second coding block includes: inter-layer prediction, and the reference block of the second coding block is a reconstructed block in the first reconstructed image with the same position as the second CU corresponding to the second coding block; when the second CU corresponding to the second coding block is located in the second area, the prediction mode of the second coding block includes: inter-frame prediction and intra-frame prediction; wherein the first reconstructed image and the second reconstructed image are used to determine the reconstructed image of the original image.

[0059] In one possible implementation, the base layer bitstream includes a reference frame identifier and a prediction mode, and a prediction decoding module, which is specifically used to: decode multiple first coding blocks to obtain a residual between a first image and a predicted image of the first image; determine a reference image of the base layer bitstream according to the reference frame identifier and the prediction mode in the base layer bitstream; predict multiple first coding blocks according to the reference image of the base layer bitstream to obtain a predicted image of the first image; add the predicted image of the first image to the residual decoded from the base layer bitstream to obtain a first reconstructed image.

[0060] In a possible implementation, the multiple first coding blocks are coding blocks of a first target CU in a first image, and the first target CU is a first CU located in a first region; wherein the predicted image of the first image includes prediction blocks of all first target CUs.

[0061] In a possible implementation manner, a residual between the prediction block of the first target CU and the first CU is zero.

[0062] In one possible implementation, the first reconstructed image is a reference image of a base layer bit stream and / or a reference image of an enhanced layer bit stream of a first image of other frames, and the reconstructed boundary pixels between the first area and the second area in the first reconstructed image are reconstructed pixels that have not undergone loop filtering.

[0063] In a possible implementation manner, the reconstructed pixels of the second area in the first reconstructed image are preset values.

[0064] In a possible implementation manner, when loop filtering is performed on the reconstructed pixels in the first area of ​​the first reconstructed image, the referenced reconstructed pixels do not include the reconstructed pixels in the second area.

[0065] In one possible implementation, when intra-frame prediction is performed on the coding block of the first target CU among multiple first coding blocks, the reference pixels of the coding block of the first target CU do not include reconstructed pixels within the second area, and the first target CU is the first CU located within the second area; when inter-frame prediction is performed on the coding block of the first target CU, the reference motion information of the coding block of the first target CU does not include motion information from the second area.

[0066] In one possible implementation, the enhancement layer bitstream includes a reference frame identifier and a prediction mode, and a prediction decoding module, which is specifically used to: decode multiple second coding blocks to obtain a residual between a second image and a predicted image of the second image; determine a reference image of the enhancement layer bitstream according to the reference frame identifier and the prediction mode in the enhancement layer bitstream; predict multiple second coding blocks according to the reference image of the enhancement layer bitstream to obtain a predicted image of the second image; add the predicted image of the second image to the residual decoded from the enhancement layer bitstream to obtain a second reconstructed image.

[0067] In a possible implementation, the multiple second coding blocks are coding blocks of a second target CU in the second image, and the second target CU is a second CU located in the second region; wherein the predicted image of the second image includes prediction blocks of all second target CUs.

[0068] In one possible implementation, the second reconstructed image is a reconstructed image of the second image, the second reconstructed image is a reference image of the enhancement layer bit stream of the second image of other frames, the pixel value of the reconstructed block of the coding block of the third target CU in the multiple second coding blocks is equal to the pixel value of the reconstructed block with the same position as the third target CU in the first reconstructed image; the third target CU is the second CU located in the first area, the image of the second area in the second reconstructed image is an image that has undergone loop filtering, and the image of the first area in the second reconstructed image is an image that has not undergone loop filtering.

[0069] In a possible implementation manner, the second image further includes the image of the first area in the original image; or the second image does not include the image of the first area in the original image.

[0070] In a possible implementation, the bit stream further includes encoded position indication information, where the position indication information is used to indicate the position of the second region; the prediction decoding module is further used to determine the second region in the first reconstructed image according to the position indication information.

[0071] In combination with the above solution, the prediction decoding module is further used to use the second reconstructed image as the reconstructed image of the original image.

[0072] In a fifth aspect, the present application provides a layered coding device, which includes: one or more processors; a memory for storing one or more computer programs or instructions; when the one or more computer programs or instructions are executed by one or more processors, the one or more processors implement a method as described in any one of the first aspects.

[0073] In a sixth aspect, the present application provides a layered coding device, comprising a processor for executing a method as described in any one of the first aspects.

[0074] In the seventh aspect, the present application provides a layered decoding device, which includes: one or more processors; a memory for storing one or more computer programs or instructions; when the one or more computer programs or instructions are executed by one or more processors, the one or more processors implement a method as described in any one of the second aspects.

[0075] In an eighth aspect, the present application provides a layered decoding device, comprising a processor for executing a method as described in any one of the second aspects.

[0076] In a ninth aspect, the present application provides a layered coding and decoding device, comprising: a processing circuit and an interface circuit; wherein the interface circuit is used to couple with a memory outside the coding and decoding device and provide a communication interface for the processing circuit to access the memory; the processing circuit is used to execute program instructions in the memory to implement a method as described in any one of the first and second aspects.

[0077] In the specific implementation process, the codec device can be a chip, the input circuit can be an input pin, the output circuit can be an output pin, and the processing circuit can be a transistor, a gate circuit, a trigger, and various logic circuits. The input signal received by the input circuit can be, for example, but not limited to, received and input by a receiver, and the signal output by the output circuit can be, for example, but not limited to, output to a transmitter and transmitted by the transmitter, and the input circuit and the output circuit can be the same circuit, which is used as an input circuit and an output circuit at different times. The embodiment of the present application does not limit the specific implementation of the processor and various circuits.

[0078] In one implementation, the encoding and decoding device may be a wireless communication device, that is, a computer device that supports wireless communication functions. Specifically, the wireless communication device may be a terminal such as a smart phone, or a wireless access network device such as a base station. The network chip may also be referred to as a system on chip (SoC), or simply as an SoC chip. The communication chip may include a baseband processing chip and a radio frequency processing chip. The baseband processing chip is sometimes also referred to as a modem or a baseband chip. The radio frequency processing chip is sometimes also referred to as a radio frequency transceiver or a radio frequency chip. In a physical implementation, some or all of the chips in the communication chip may be integrated inside the SoC chip. For example, the baseband processing chip is integrated in the SoC chip, and the radio frequency processing chip is not integrated with the SoC chip. The interface circuit may be a radio frequency processing chip in a wireless communication device, and the processing circuit may be a baseband processing chip in a wireless communication device.

[0079] In another implementation, the codec device may be a part of a wireless communication device, such as an integrated circuit product such as a network chip or a communication chip. The interface circuit may be an input / output interface, an interface circuit, an output circuit, an input circuit, a pin or a related circuit on the chip or chip network. The processor may also be embodied as a processing circuit or a logic circuit.

[0080] In a tenth aspect, the present application provides a computer-readable storage medium, in which a program code is stored. When the program code is executed by a processor, the method as described in any one of the first and second aspects is implemented.

[0081] In an eleventh aspect, the present application provides a chip, comprising: at least one processor. The at least one processor is configured to execute the method as described in any one of the first aspect and the second aspect.

[0082] Optionally, the chip further includes a memory. At least one processor is used to execute the code in the memory, and when the at least one processor executes the code, the chip implements the method as described in any one of the first aspect and the second aspect.

[0083] Optionally, the above chip may also be an integrated circuit.

[0084] In a twelfth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to implement the method as described in any one of the first and second aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1A A schematic diagram of an application scenario provided for an embodiment of the present application;

[0086] Figure 1B A schematic diagram of a compression framework provided in an embodiment of the present application;

[0087] Figure 2A Another schematic diagram of a compression framework provided in an embodiment of the present application;

[0088] Figure 2B A schematic diagram of an original image, a first image, and a second image provided in an embodiment of the present application;

[0089] Figure 3A A schematic diagram of a process of a layered encoding method provided in an embodiment of the present application;

[0090] Figure 3B A schematic diagram of a bit stream structure provided in an embodiment of the present application;

[0091] Figure 4A A schematic diagram of a process of a layered decoding method provided in an embodiment of the present application;

[0092] Figure 4B A schematic diagram of a first reconstructed image and a second reconstructed image provided in an embodiment of the present application;

[0093] Figure 5A A schematic diagram of another hierarchical coding method provided in an embodiment of the present application;

[0094] Figure 5B A schematic diagram of a hierarchical coding process provided in an embodiment of the present application;

[0095] Figure 5C A prediction reference schematic diagram provided for an embodiment of the present application;

[0096] Fig. 6A A schematic diagram of another hierarchical decoding method provided in an embodiment of the present application;

[0097] Figure 6B A schematic diagram of a hierarchical decoding process provided in an embodiment of the present application;

[0098] Fig. 7A A schematic diagram of a compression process provided in an embodiment of the present application;

[0099] Figure 7B A schematic diagram of a compression process provided in an embodiment of the present application;

[0100] Fig. 8A A block diagram of a layered coding device provided in an embodiment of the present application;

[0101] Figure 8B A block diagram of another hierarchical coding device provided in an embodiment of the present application;

[0102] Fig. 9 A block diagram of a layered decoding device provided in an embodiment of the present application;

[0103] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0104] Fig.11 A schematic diagram of the structure of a layered encoding and decoding device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0105] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0106] The terms "first", "second", etc. in the specification embodiments, claims, and drawings of the present application are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. Methods, networks, products, or devices are not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0107] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0108] Please refer to Figure 1A , Figure 1A A schematic diagram of an application scenario provided for an embodiment of the present application. Figure 1A The application scenario shown is a video surveillance scenario; it should be understood that the embodiments of the present application can also be applied to any scenario that requires video (or image) privacy protection, and the embodiments of the present application do not limit the application scenario.

[0109] Reference Figure 1A , for example, image acquisition devices can be deployed at intersections, shopping mall entrances and exits, school gates and other locations, and the image acquisition devices can collect video data, and then transmit the collected video data to at least one terminal device through the network. For example, the video data collected by the image acquisition device deployed at the intersection can be transmitted to the smart screen, the video data collected by the image acquisition device deployed at the shopping mall entrance and exit can be transmitted to the smart screen and tablet computer, the video data collected by the image acquisition device deployed at the school gate can be transmitted to the smart screen and personal computer, and so on. After the terminal device receives the video data sent by the image acquisition device, it can store and display the video data (or the video data processed as required) to facilitate the user to detect the monitored area.

[0110] It should be noted that in the embodiment of the present application, the actions of the image acquisition device to collect video data, transmit video data, and the terminal device to store, process and display video data are all carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0111] Exemplarily, the image acquisition device can compress (or encode) the acquired video data to obtain a bit stream (Bit stream; where the bit stream can also be called a code stream or bit stream), and then transmit the bit stream to the terminal device to reduce the amount of transmitted data and reduce bandwidth requirements.

[0112] Please refer to Figure 1B , Figure 1B A schematic diagram of a compression framework provided for an embodiment of the present application.

[0113] Reference Figure 1B , exemplarily, the image acquisition device may include a camera, an encoding module (or encoder) and a sending module. , exemplarily, the encoding module may be a software module or a hardware module, which is not limited in the present application embodiment. It should be understood that, Figure 1B This is only an example of an image acquisition device. In other embodiments of the present application, the image acquisition device has a Figure 1B More modules are shown, and the embodiments of the present application are not limited to this.

[0114] Reference Figure 1B , exemplarily, the terminal device may include a display module, a decoding module (or decoder) and a receiving module. Exemplarily, the decoding module may be a software module or a hardware module, which is not limited in the present application embodiment. It should be understood that, Figure 1B This is only an example of a terminal device. The terminal devices in other embodiments of the present application have Figure 1B More modules are shown, and the embodiments of the present application are not limited to this.

[0115] Continue to refer to Figure 1B , exemplarily, the process of an image acquisition device sending the raw video data collected by a camera to a terminal device for display is as follows: the camera can output the collected raw video data to the encoding module; then, the encoding module can encode the raw video data to obtain a bit stream, and output the encoded bit stream to the sending module; then, the sending module can send the bit stream to the terminal device. Subsequently, the receiving module of the terminal device can receive the bit stream; then, the bit stream is output to the decoding module; then, the decoding module can decode the bit stream to obtain reconstructed video data and output the reconstructed video data to the display module, and the display module displays the reconstructed video data.

[0116] For example, the following Figure 1B The image acquisition device is called the encoding end, and can Figure 1B The terminal device is called a decoding end; the encoding process of the encoding end and the decoding process of the decoding end can refer to the description of the subsequent embodiments.

[0117] First, some of the nouns in the encoding process are explained:

[0118] Block-based video coding: Before encoding the original image, the entire image needs to be divided into many coding units (CUs) according to certain rules. The size of the CU can generally support 4×4 to 64×64.

[0119] CU prediction coding: After the CU is divided, each CU is controlled to choose intra-frame prediction or inter-frame prediction according to the coding parameters.

[0120] Intra-frame prediction: Use the spatial pixel correlation within the current frame to remove information redundancy in the spatial domain. Specifically, when encoding the current CU within the frame, refer to the reconstructed pixels of its spatial neighbors to predict and estimate the approximate pixel values ​​for the current CU.

[0121] Inter-frame prediction: Use the correlation between the current frame and the previously encoded / decoded frame in the time domain to remove information redundancy in the time domain. Specifically, when the current CU is inter-coded, it refers to the reconstructed pixels in the previously encoded / decoded frame and predicts and estimates the approximate pixel values ​​through MV.

[0122] Transform quantization: After prediction, the CU obtains a pixel value close to the original image (i.e., the predicted value), and then the original pixel value at the current CU is subtracted from the predicted value to obtain the pixel residual value of the current CU, and then the residual is transformed to obtain the transform coefficient, so as to map the spatial domain residual to the frequency domain coefficient. The size of the transform coefficient is further reduced through quantization.

[0123] Entropy coding: The last step is to entropy encode the transformed quantized coefficients, motion information and control parameters to obtain the final compressed bit stream. The motion information includes motion vector (MV) and so on.

[0124] Pixel reconstruction: The reconstruction process of general video coding refers to the process of adding pixel residuals to predicted pixels to obtain a reconstructed image. The pixel residuals used here are generally the residuals after inverse quantization / inverse transformation distortion. If the residual distortion is too large, the distortion between the reconstructed image and the original image will be greater. The reconstruction process exists at the encoding end and the decoding end at the same time, and the operation is exactly the same. In addition to being played and displayed, the reconstructed image also needs to be used as a reference image for inter-frame prediction.

[0125] Layered video coding: Divide the video into different layers in different dimensions (frame rate, resolution, quality, etc.), each layer corresponds to a different level (the bottom layer is the base layer, and the layers above are enhanced layers), and encode them separately to output multiple layers of bitstreams. The characteristic of layered coding is that the bitstream is layered. The base layer (BL) bitstream can be decoded separately to output the base layer video; the enhancement layer (EL) bitstream depends on the base layer data to be decoded, and the enhanced layer video is output after decoding. Compared with the base layer video, the enhancement layer video has higher specifications in terms of quality, resolution or frame rate.

[0126] Base layer reconstructed image: The base layer bitstream can be decoded independently, and after decoding, it generally outputs an image of lower specifications (such as low frame rate, low resolution or low quality). In privacy coding, this low-specification image is a non-privacy image that does not contain a privacy area, and is referred to as a base layer reconstructed image (also referred to as a BL reconstructed image) in this invention document.

[0127] Enhanced layer reconstructed image: The enhanced layer bitstream contains additional information required to reconstruct a high-specification image. Its decoding depends on the base layer data. After decoding, it outputs a reconstructed image with a higher specification (high frame rate, high resolution or high quality) than the base layer image.

[0128] Picture order count (POC): It is actually the display order of the pictures, that is, the picture number in the picture playback order.

[0129] Please refer to Figure 2A , Figure 2A Another compression framework schematic diagram provided in an embodiment of the present application, Figure 2A A framework for layered video coding is shown. Figure 2A The dashed lines in the figure are optional operations.

[0130] Reference Figure 2A , exemplarily, before encoding at the encoding end, the original image may be preprocessed to obtain a first image and a second image.

[0131] Exemplarily, the original image may refer to an image that is originally expected to be transmitted to the remote end. The original image may be generated in a variety of ways, for example, the original image may be an image captured by an image acquisition device, an image received from other terminal devices, an image generated by AI (Artificial Intelligence), an image processed by an image processing tool, etc., and this application does not limit this.

[0132] Exemplarily, the original image may include user information; wherein user information may refer to information that directly or indirectly describes the identity of the user. For example, user information includes, but is not limited to: user name, date of birth, ID number, address, telephone number, face, license plate number, sports posture, clothing, etc., which is not limited in this application. In some scenarios, user information may also be referred to as private information, which may refer to secrets that are not willing to be disclosed or known to others (people outside a certain range), and this secret is irrelevant to other people and social interests.

[0133] Specifically, the process of preprocessing the original image to obtain the first image and the second image may be as follows:

[0134] First, user information detection can be performed on the original image to determine the area where the user information is located. For ease of explanation, the area containing user information in the original image (that is, the area where the user information is located) is called the second area, and the area not containing user information in the original image is called the first area; it should be understood that the first area in the original image is the area in the original image other than the second area. The method for detecting user information on the original image can be, for example, an intelligent detection method based on AI, and this application does not limit this.

[0135] For example, the original image can be Figure 2B As shown in (1), the second region can be Figure 2B As shown in the dotted box in (1), the first area is Figure 2B (1) except the second area in the original image.

[0136] While determining the area where the user information is located, position indication information corresponding to the original image can also be obtained, and the position indication information is used to indicate the position of the second area.

[0137] Next, generate a second image. In one possible way, the original image can be used as the second image; that is, the second image includes an image of the first area in the original image and an image of the second area in the original image. In one possible way, the pixel values ​​of the pixels in the second area in the original image can be kept unchanged, and the pixel values ​​of the pixels in the first area in the original image can be replaced. For example, the replacement can be performed in at least one of the following ways: solid color filling, mosaic, or Gaussian blur, etc., which is not limited in this embodiment of the present application. For example, if the replacement method is solid color filling, the second image can be as follows: Figure 2B (2) as shown; Figure 2B In (2), the pixel values ​​of the pixels in the second area of ​​the original image remain unchanged, and the pixel values ​​of the pixels in the first area of ​​the original image are filled with white (it should be understood that other colors can also be used for filling, and this application does not limit this).

[0138] And, generate a first image. Exemplarily, the pixel values ​​of the pixels in the first area of ​​the original image can be kept unchanged, and the pixel values ​​of the pixels in the second area of ​​the original image can be replaced. For example, the replacement can be performed in at least one of the following ways: solid color filling, mosaic or Gaussian blur, etc., which is not limited in this application. For example, if the replacement method is solid color filling, the first image can be as follows Figure 2B (3) as shown; Figure 2B In (3), the pixel values ​​of the pixels in the first area of ​​the original image remain unchanged, and the pixel values ​​of the pixels in the second area of ​​the original image are filled with white (it should be understood that other colors can also be used for filling, and this application does not limit this).

[0139] That is, the first image includes the image of the first area in the original image and does not include the image of the second area in the original image, the second image includes the image of the second area in the original image, and the size of the first image and the size of the second image are the same as the size of the original image.

[0140] It should be noted that the preprocessing operation can be performed by the encoder or by other modules, and the present application embodiment does not limit this. The present application takes the preprocessing performed by other modules as an example for explanation.

[0141] Afterwards, the first image and the second image can be input into the same encoder for encoding, as shown below: Figure 3A Description in the examples.

[0142] Please refer to Figure 3A , Figure 3A A schematic diagram of a process of a layered encoding method provided in an embodiment of the present application, which can be applied to an encoder.

[0143] 301. Acquire a first image and a second image.

[0144] The first image includes an image of a first area in the original image and does not include an image of a second area in the original image. The second image includes an image of the second area in the original image. The size of the first image and the size of the second image are the same as the size of the original image. The image of the second area in the original image contains user information.

[0145] 302. Divide the first image to obtain multiple first CUs.

[0146] Any first CU among the plurality of first CUs is located within the first region or within the second region. That is, each first CU is not divided by the boundary line of the privacy region. In a first CU, all pixels are within the first region, or all pixels are within the second region.

[0147] 303. Perform prediction encoding of a base layer on the multiple first CUs to obtain a base layer bit stream.

[0148] The prediction mode of the first CU includes: inter-frame prediction and intra-frame prediction. When inter-frame prediction is performed on the first CU, if the reference pixel of the first CU is located in the second area of ​​the reference image, the pixel value of the reference pixel of the first CU adopts the preset pixel value, and the preset pixel value is used to predict the first CU to obtain the predicted pixel value of the first CU. If the reference motion information of the first CU comes from the second area, the reference motion information is set to invalid.

[0149] When inter-frame prediction is performed on the first CU, the motion information of the first CU needs to be obtained first. The motion information may include a motion vector and a reference frame index, etc. For example, the motion information may be directly inherited from the first CU's neighboring CU, or the motion information may be obtained by searching for a matching reference block in the reference image through motion estimation. Then, motion compensation is required to obtain the prediction value of the first CU based on the reference pixel.

[0150] If the motion information is obtained from the second area in the first image, or the reference block is located in the second area in the reference image, the motion information is set to be invalid and unavailable.

[0151] The reference image is the first reconstructed image of the encoded frame before the current frame. The preset pixel value can be, for example, the median value or the value transmitted through the image header of the first image. As long as the preset pixel values ​​of the encoder and decoder are the same, the embodiment of the present application does not limit this.

[0152] When encoding multiple first CUs, a reconstructed block of each first CU in the multiple first CUs can also be generated to obtain a first reconstructed image. The first reconstructed image is a reference image of the first images of other frames and / or a reference image of the second image. The reconstructed pixels within the first area in the first reconstructed image are reconstructed pixels that have undergone loop filtering, and the reconstructed pixels within the second area are reconstructed pixels that have not undergone loop filtering. The reconstructed boundary pixels between the first area and the second area in the first reconstructed image are reconstructed pixels that have not undergone loop filtering. The boundary between the first area and the second area can be determined by position indication information.

[0153] 304. Divide the second image to obtain multiple second CUs.

[0154] Any second CU among the plurality of second CUs is located within the first region or within the second region. That is, each second CU is not divided by the boundary line of the privacy region. In a second CU, all pixels are within the first region, or all pixels are within the second region.

[0155] Exemplarily, the second image is divided in the same manner as the first image, that is, each second CU has a first CU at the same position in the first image.

[0156] 305. Perform prediction coding of enhancement layers on multiple second CUs to obtain enhancement layer bit streams.

[0157] Among them, the base layer bit stream and the enhancement layer bit stream are interwoven into one bit stream.

[0158] When the second CU is located in the first region, the prediction mode of the second CU includes: inter-layer prediction, and the reference block of the second CU is a reconstructed block in the first reconstructed image at the same position as the second CU, and the first reconstructed image is a reconstructed image of the first image. When the second CU is located in the second region, the prediction mode of the second CU includes: inter-frame prediction and intra-frame prediction.

[0159] In the embodiment of the present application, the second CU located in the first area is referred to as the third target CU, and the motion vector of the third target CU is always (0, 0). After obtaining the prediction block of the third target CU, the residual between the prediction block of the third target CU and the third CU is set to 0. That is, the reconstructed block of the third target CU directly copies the pixels at the same position as the third target CU in the first reconstructed image of the BL layer.

[0160] Exemplarily, first, multiple first CUs may be used as images of a base layer and input into an encoder, and the encoder may encode the multiple first CUs at the base layer to obtain a base layer bit stream (hereinafter referred to as a BL bit stream). Next, multiple second CUs may be used as images of an enhancement layer and input into the encoder, and the encoder may encode the multiple second CUs at the enhancement layer to obtain an enhancement layer bit stream (hereinafter referred to as an EL bit stream). That is, the encoder first executes 303 and then executes 304.

[0161] It should be noted that the multiple first CUs and the multiple second CUs are encoded by the same encoder, wherein one encoder outputs one bit stream.

[0162] In this way, after the encoder completes encoding of the multiple first CUs and the multiple second CUs, the BL bit stream and the EL bit stream can be interleaved into one bit stream (this bit stream can also be called the bit stream of the original image). The one bit stream can include the BL bit stream and the EL bit stream, and the BL bit stream is located before the EL bit stream.

[0163] Exemplarily, for video data including n (n is a positive integer) frames of original images, the above steps 301 to 305 can be performed for each frame of original image to obtain a bit stream of n frames of original images (hereinafter referred to as n-frame bit stream). Among them, the first image and the second image generated based on the i-th (i is an integer between 1 and n) frame of original image can be referred to as the i-th frame first image and the i-th frame second image. In this way, a bit stream output by the encoder can include n-frame bit streams, and each frame of the bit stream includes an EL bit stream and a BL bit stream, and the BL bit stream is located before the EL bit stream; Figure 3B shown.

[0164] Figure 3B A schematic diagram of a bit stream structure provided in an embodiment of the present application. Figure 3BIn the embodiment, a bit stream output by the encoder includes n frame bit streams: the first frame bit stream (i.e., the bit stream of the first frame original image) includes the BL_1 bit stream and the EL_1 bit stream, and the BL_1 bit stream is before the EL_1 bit stream; the second frame bit stream (i.e., the bit stream of the second frame original image) includes the BL_2 bit stream and the EL_2 bit stream, and the BL_2 bit stream is before the EL_2 bit stream; ...; the n frame bit stream (i.e., the bit stream of the n frame original image) includes the BL_n bit stream and the EL_n bit stream, and the BL_n bit stream is before the EL_n bit stream.

[0165] Exemplarily, the header of the BL bitstream output by the encoder includes a frame identifier (such as POC) and a layer identifier (Layer ID); and the header of the EL bitstream includes a frame identifier and a layer identifier. The frame identifier is used to identify a frame bitstream, and the layer identifier is used to identify the BL bitstream and the EL bitstream in each frame bitstream. It should be understood that the value of the frame identifier in each frame bitstream is different, and the value of the layer identifier of the BL bitstream and the EL bitstream in each frame bitstream is different.

[0166] Exemplarily, for the first image and the second image generated based on the same frame of original image, the values ​​of the frame identifiers in the BL bitstream and the EL bitstream are the same; therefore, the header of the EL bitstream may not include the frame identifier, which is not limited in this embodiment of the present application.

[0167] It should be noted that the EL bit stream is output after the BL bit stream, that is, after the encoder outputs the EL bit stream, the EL bit stream is connected to the BL bit stream. At this time, the EL bit stream and the BL bit stream are interleaved to form a bit stream (it can be understood that the BL bit stream and the EL bit stream are interleaved in the bit stream output by the encoder); it is not that the encoder performs additional operations (such as interleaving operations) to make the EL bit stream and the BL bit stream interleave to form a bit stream.

[0168] Based on the above, it can be seen that the embodiment of the present application can be regarded as a layered encoding of the original image, wherein, in the layered encoding process of the original image, the encoding of the base layer corresponds to the encoded image of the base layer as the first image, and the encoding of the enhancement layer corresponds to the encoded image of the enhancement layer as the second image.

[0169] When encoding multiple second CUs, a reconstructed block of each second CU in the multiple second CUs can also be generated to obtain a second reconstructed image. The second reconstructed image is a reference image of the second image of other frames, and the pixel value of the reconstructed block of the third target CU in the multiple second CUs is equal to the pixel value of the reconstructed block in the first reconstructed image at the same position as the third target CU. The third target CU is a second CU located in the first area, the image of the second area in the second reconstructed image is an image that has been loop filtered, and the image of the first area in the second reconstructed image is an image that has not been loop filtered.

[0170] Since the reference queue for encoding the enhancement layer includes the reconstructed image of the base layer (that is, the reconstructed image of the first image (hereinafter referred to as the first reconstructed image)), that is to say, the reference queue of the second image includes the first reconstructed image; compared with the prior art in which the reference queue of the second image (that is, the privacy image) only includes the reconstructed block of the second image of the current frame and the second reconstructed images of other frames (that is, the reconstructed images of the second images of other frames), the embodiment of the present application encodes more reference images for the second image. In this way, the embodiment of the present application can obtain a more accurate prediction block, and thus can improve the encoding performance of the enhancement layer (improve encoding quality or reduce bit rate overhead), thereby improving the overall encoding performance. In addition, the present application encodes the first image and the second image, and obtains a bit stream, which does not require bit stream synchronization operations, and can reduce computational overhead.

[0171] And the following four restrictions can ensure that the reconstructed pixel values ​​(after loop filtering) and motion information stored in the DPB in the first area of ​​the second image are completely consistent with the first image: 1. The second CU located in the second area of ​​the second image can only refer to the reconstructed block at the same position in the first reconstructed image for inter-layer prediction. Intra-frame prediction and inter-frame prediction are not allowed, and the residual during inter-layer prediction is set to 0. 2. Any first CU in each first CU is located in the first area or the second area, and any second CU in each second CU is located in the first area or the second area. 3. When the first CU derives the inter-frame prediction motion vector, if the time domain motion information used comes from the second area, the time domain motion information is set to be unavailable. 4. The reconstructed pixels located in the first area of ​​the second image are not loop filtered, and the reconstructed pixels located in the second area are loop filtered, and the filtering of the reconstructed pixels at the boundary between the first area and the second area can refer to the reconstructed pixels in the first area.

[0172] The following two restrictions allow the codec to obtain consistent pixel values ​​even if the image in the second region of the first image does not exist: 5. The second CU in the second region of the second image can only perform time domain prediction or intra-frame prediction, and inter-layer prediction is not allowed. 6. When the first CU in the first image performs inter-frame prediction motion compensation, if the reference pixel of the first CU is located in the second region, the pixel value of the reference pixel is replaced by a preset pixel value.

[0173] Through the above six restrictions, BL and EL can reuse one DPB, which solves the problem of high encoding and decoding costs caused by doubling of DPB resources in layered coding, thereby further reducing the encoding and decoding costs.

[0174] Optionally, when filtering at the BL layer, in order to prevent the filled privacy area value from polluting the non-privacy area, the boundary pixels between the non-privacy area and the privacy area are not filtered, wherein the privacy area boundary is determined according to the privacy area indication information.

[0175] All CUs in the entire image cannot be divided by the boundaries of the privacy region, and the division of the privacy region indication information must ensure that it meets this constraint. That is, in a coding block, either all pixels are in the privacy region or all pixels are in the non-privacy region.

[0176] Exemplarily, after the encoding end obtains the bit stream of the original image, the EL bit stream in the bit stream of the original image can be encrypted to protect the user information in the original image during the process of transmitting the original image to the remote end, thereby ensuring the security of the user information in the original image. The encryption operation can be performed by the encoding end system layer.

[0177] Exemplarily, after receiving the bit stream, the decoding end may input the BL bit stream in the bit stream into the decoder; on the other hand, the encrypted EL bit stream in the bit stream may be decrypted and input into the decoder; wherein the decryption operation may be performed by the system layer of the decoding end. Next, referring to Figure 2A , the decoder decodes the BL bit stream and the EL bit stream to obtain a reconstructed image of the first image (hereinafter referred to as the first reconstructed image) and a reconstructed image of the second image (hereinafter referred to as the second reconstructed image). After that, the post-processing module performs post-processing based on the first reconstructed image and the second reconstructed image to obtain a reconstructed image of the original image. The decoding process of the decoder can be as follows Figure 4A As shown in the embodiment.

[0178] Please refer to Figure 4A , Figure 4A A schematic diagram of the process of a layered decoding method provided in an embodiment of the present application, which can be applied to a decoder.

[0179] 401. Receive a bit stream, where the bit stream includes a base layer bit stream and an enhancement layer bit stream.

[0180] The base layer bitstream includes a plurality of first coding blocks, and the enhancement layer bitstream includes a plurality of second coding blocks, wherein the first coding blocks are coding blocks of the first CU, and the second coding blocks are coding blocks of the second CU.

[0181] Exemplarily, after receiving the bit stream, the decoder can read the frame identifier from the bit stream and determine a frame bit stream according to the frame identifier. Then, the layer identifier can be read from the frame bit stream to determine the BL bit stream and the EL bit stream.

[0182] 402. Perform prediction decoding on a plurality of first coding blocks at a base layer to obtain a first reconstructed image.

[0183] The first reconstructed image is a reconstructed image of the first image, the first image includes an image of a first area in the original image and does not include an image of a second area in the original image, and the image of the second area in the original image includes user information.

[0184] The prediction mode of the first coding block includes: inter-frame prediction and intra-frame prediction. When inter-frame prediction is performed on the first coding block, if the reference pixel of the bit stream of the first coding block is located in the second area of ​​the reference image, the pixel value of the reference pixel of the first coding block adopts a preset pixel value; if the reference motion information of the first coding block comes from the second area, the reference motion information is set to be invalid.

[0185] The decoder can decode the base layer bit stream to obtain the first reconstructed image. Figure 2B (3) As shown in FIG. Figure 4B (1) as shown.

[0186] The first reconstructed image is a reference image of the BL bitstream of other frames and / or a reference image of the EL bitstream. The reconstructed pixels within the first area in the first reconstructed image are reconstructed pixels that have undergone loop filtering, and the reconstructed pixels within the second area are reconstructed pixels that have not undergone loop filtering. The reconstructed boundary pixels between the first area and the second area in the first reconstructed image are reconstructed pixels that have not undergone loop filtering. The boundary between the first area and the second area can be determined by position indication information.

[0187] 403. Perform prediction decoding of the enhancement layer on the multiple second coding blocks to obtain a second reconstructed image.

[0188] The first reconstructed image and the second reconstructed image are used to determine a reconstructed image of the original image.

[0189] Among them, the second reconstructed image is a reconstructed image of the second image, the second image includes an image of the second area in the original image, and the size of the first image and the size of the second image are the same as the size of the original image. When the second CU corresponding to the second coding block is located in the first area, the prediction mode of the second coding block includes: inter-layer prediction, and the reference block of the second coding block is a reconstructed block in the first reconstructed image with the same position as the second CU corresponding to the second coding block; when the second CU corresponding to the second coding block is located in the second area, the prediction mode of the second coding block includes: inter-frame prediction and intra-frame prediction. It should be noted that because the first reconstructed image and the second reconstructed image share a DPB, and the reconstructed pixels of the second CU located in the first area in the second image are the same as the reconstructed pixels of the first CU located in the first area in the first image, in one implementation, the decoding of the coding block of the second CU located in the first area in the second image can be skipped to reduce the decoding complexity.

[0190] The decoder can decode the enhancement layer bit stream to obtain a second reconstructed image. For example, when the second image is the original image (such as Figure 2B (1)), the second reconstructed image can be Figure 4B (2) As shown in FIG. 2 . When the second image includes the image of the second region in the original image and does not include the image of the first region in the original image (such as Figure 2B (2)) When the second reconstructed image is Figure 4B (3) as shown.

[0191] It should be noted that when the encoder performs lossless encoding on the first image and the second image, Figure 4B The first reconstructed image in (1) is Figure 2B The first image in (3) is the same, Figure 4B The second reconstructed image in (2) is Figure 2B The original image in (1) is the same as Figure 4B The second reconstructed image in (3) is Figure 2B (2) is the same as the second image. When the encoder performs lossy encoding on the first image and the second image, Figure 4B The first reconstructed image in (1) is Figure 2B There is a difference in the first image in (3), Figure 4B The second reconstructed image in (2) is Figure 2B There are differences in the original image in (1). Figure 4B The second reconstructed image in (3) is Figure 2B There is a difference in the second image in (2).

[0192] It should be understood that Figure 4B It is only a schematic diagram of the first reconstructed image and the second reconstructed image, and does not mean that the encoding and decoding method of the embodiment of the present application is used for Figure 2B The real first reconstructed image and the second reconstructed image are obtained after encoding and decoding the first image and the second image.

[0193] Refer again Figure 2A , exemplarily, when the second image includes an image of the second area in the original image and does not include an image of the first area in the original image, the first area in the first reconstructed image and the second area in the second reconstructed image can be determined; then, the image of the first area in the first reconstructed image and the image of the second area in the second reconstructed image are fused to obtain a reconstructed image of the original image. For example, the pixel value of a pixel point in the second area in the first reconstructed image can be replaced with the pixel value of a pixel point at a corresponding position in the second area in the second reconstructed image. For another example, the pixel value of a pixel point in the first area in the second reconstructed image can be replaced with the pixel value of a pixel point at a corresponding position in the first area in the first reconstructed image.

[0194] The second reconstructed image is a reference image of the EL bit stream of other frames, and the pixel values ​​of the reconstructed pixels within the first area in the second reconstructed image are the same as the pixel values ​​of the reconstructed pixels at the same position in the first reconstructed image. The image of the second area in the second reconstructed image is an image that has been loop filtered, and the image of the first area in the second reconstructed image is an image that has not been loop filtered.

[0195] Exemplarily, when the second image is an original image, the second reconstructed image may be used as a reconstructed image of the original image.

[0196] The encoding process of the base layer and the encoding process of the enhancement layer are described below.

[0197] Please refer to Figure 5A , Figure 5A A process diagram of another encoding method provided in an embodiment of the present application. Figure 5A In the process of enhancement layer encoding, inter-frame prediction and / or intra-frame prediction are performed on the CU belonging to the second area in the second image, and inter-layer prediction is performed on the CU that does not belong to the second area (ie, belongs to the first area) in the second image.

[0198] 501. Acquire a first image and a second image, wherein the first image includes an image of a first area in an original image and does not include an image of a second area in the original image, and the second image includes an image of the second area in the original image. A size of the first image and a size of the second image are the same as a size of the original image, and the image of the second area in the original image contains user information.

[0199] This process may refer to the aforementioned process 301, and the embodiment of the present application will not be described in detail here.

[0200] 502. Acquire location indication information, where the location indication information is used to indicate the location of the second area.

[0201] In one possible manner, when the second area is a rectangle, the position indication information may include the coordinates of the upper left corner of the second area and the size of the second area. When the second area is a circle, the position indication information may include the coordinates of the center of the second area and the radius of the second area. It should be understood that when the second area is other shapes, the position indication information may include other information, and the present application does not limit this.

[0202] In one possible way, the position indication information may be an image block identification map. Specifically, the original image may be divided into m (m is a positive integer) image blocks of a fixed size (which may be set as required, such as 2*2); then, a preset identifier is used for each image block to identify whether it is in the second area; for example, for an image block, when the image block is in the second area, the value of the preset identifier of the image block is set to the first identifier value; when the image block is not in the second area, the value of the preset identifier of the image block is set to the second identifier value; in this way, an image block identification map may be obtained; wherein the image block identification map may include m pixels, one pixel corresponds to one image block, and the pixel value of each pixel is the value of the preset identifier. Wherein, the first identification value is different from the second identification value, and may be set specifically as required, and the present application does not limit this. For example, the first identification value is "1" and the second identification value is "0"; at this time, among the m pixels contained in the image block identification map, the pixel values ​​of m1 pixels are "1" and the pixel values ​​of m2 pixels are "0". Wherein, m1+m2=m, m1 and m2 are integers greater than or equal to 0.

[0203] It should be understood that the image block identification map may also be referred to as an image block identification matrix; the image block identification matrix may include m elements, one element corresponds to one image block, and the value of each element is a preset identification value.

[0204] In a possible manner, the above-mentioned image block may be a CU.

[0205] In a possible manner, the above-mentioned image block includes only one pixel. In this case, the position indication information may be a pixel identification map.

[0206] It should be noted that the location indication information may also be expressed in other forms, and this application does not impose any limitation on this.

[0207] 503. Divide the first image to obtain multiple first CUs.

[0208] 504. Perform inter-frame prediction or intra-frame prediction on multiple first CUs to obtain a predicted image of the first image.

[0209] Optionally, encoding of the first CU located in the second region may be skipped, that is, only inter-frame prediction or intra-frame prediction of BL is performed on each first target CU in the multiple first CUs.

[0210] Optionally, the reconstructed pixels in the second area can be filled in a manner agreed upon by the codec, such as being filled with a certain color value, which can be transmitted in the bitstream. By further agreeing on the reconstructed pixel filling value in the second area of ​​the BL layer at the codec end, it is only necessary to encode the content of the EL layer once, and the image in the first area can be obtained by encoding the BL layer, and the image in the second area can be obtained by encoding the EL layer. The purpose of encoding once and decoding once is achieved, further reducing the implementation cost.

[0211] For example, please refer to Figure 5B , Figure 5B A schematic diagram of a coding process provided in an embodiment of the present application, referring to Figure 5B (1), 503 may include the following steps:

[0212] 5041. Perform inter-frame prediction or intra-frame prediction of BL on the first target CU to obtain a prediction block of each first target CU.

[0213] Among them, the predicted image of the first image includes all the prediction blocks of the first target CU. When intra-frame prediction is performed on the first target CU, the reference pixels of the first target CU do not include the reconstructed pixels within the second area. When inter-frame prediction is performed on the first target CU, the reference motion information of the first target CU does not include the motion information from the second area. In this way, the encoding and decoding of the BL layer adds the restriction that the information (reconstructed pixels, prediction mode, motion information, etc.) of the second area in the current frame cannot be referenced, so that when the EL layer image is to be restored, only the coding blocks of the first CU in the first area of ​​the BL layer and the coding blocks of the second CU in the second area of ​​the EL layer need to be decoded, and only one decoding is required, thereby further reducing the implementation cost.

[0214] Exemplarily, it is possible to determine whether to perform inter-frame prediction or intra-frame prediction on the first target CU according to pre-configured encoding parameters. When it is determined that intra-frame prediction is performed on the first target CU, a search can be performed from the reconstructed block of the first image of the current frame to determine the prediction block of the first target CU. Next, the residual between the first target CU and the prediction block of the first target CU is determined.

[0215] Exemplarily, when it is determined that inter-frame prediction is to be performed on the first target CU, motion search and motion estimation can be performed from the first reconstructed image of other frames (i.e., the reconstructed image of the first image of other frames) to determine the MV of the first target CU and the prediction block of the first target CU; then, the residual between the first target CU and the prediction block of the first target CU is determined.

[0216] Exemplarily, the first target CU may be subjected to inter-frame prediction and intra-frame prediction respectively to determine two candidate prediction blocks of the third CU: candidate prediction block 1 and candidate prediction block 2, wherein candidate prediction block 1 is determined by inter-frame prediction and candidate prediction block 2 is determined by intra-frame prediction. Then, the prediction block of the first target CU may be determined based on the two candidate prediction blocks of the first target CU. In one possible manner, the best candidate prediction block is selected from candidate prediction block 1 and candidate prediction block 2 as the prediction block of the first target CU. For example, based on candidate prediction block 1 and candidate prediction block 2, the rate-distortion loss of the first image is predicted respectively (it may also be other types of losses, which are not limited in the present application), and the candidate prediction block with the smallest rate-distortion loss is selected as the best candidate prediction block.

[0217] 505. Encode a residual between the first image and a predicted image of the first image to obtain a BL bit stream.

[0218] For example, please refer to Figure 5B , Figure 5B A schematic diagram of a coding process provided in an embodiment of the present application, referring to Figure 5B (1), 504 may include the following steps:

[0219] 5051. Encode the residual between the first target CU and the prediction block of the first target CU to obtain the bit stream of the first target CU; wherein the bit streams of all first target CUs constitute the BL bit stream.

[0220] Corresponding to the first example in 5031 above, the residual is transformed, quantized, and entropy encoded to obtain the bit stream of the first target CU. Exemplarily, the residual is transformed to obtain a transform coefficient, which maps the spatial domain residual to the frequency domain coefficient to obtain the transform coefficient. The transform coefficient can then be quantized to obtain the residual coefficient; and the size of the transform coefficient can be further reduced. Next, the residual coefficient is entropy encoded to obtain the bit stream of the first target CU.

[0221] Corresponding to the second example of 5031 above, the residual is transformed, quantized and entropy encoded, and the MV of the first target CU is entropy encoded to obtain the bit stream of the first target CU.

[0222] Corresponding to the third example of 5031 above, the residual is transformed, quantized and entropy encoded, and the MV of the third CU is entropy encoded to obtain the bit stream of the third CU.

[0223] It should be noted that the operations performed in 5041 for encoding the residual between the first target CU and the prediction block of the first target CU include but are not limited to: transformation, quantization, entropy coding, etc. For example, the residual between the first target CU and the prediction block of the first target CU may be transformed and quantized to obtain the residual coefficient of the first target CU; then, the residual coefficient of the first target CU may be entropy coded to obtain the bit stream of the first target CU.

[0224] Exemplarily, a reconstruction operation may be performed during the encoding of the first image to generate a first reconstructed image, wherein the first reconstructed image may be used as a reference image of the first image of other frames and / or a reference image of the second image. The first reconstructed image may also be referred to as a reconstructed image of the base layer.

[0225] Reference Figure 5B (1) Exemplarily, the residual coefficients of the first target CU can be dequantized and inversely transformed to obtain the residual of the first target CU (wherein, the residual obtained by dequantization and inverse transformation is different from the residual before transformation and quantization, and the present application does not distinguish between the two in terms of name); then, the residual of the first target CU is added to the prediction block of the first target CU to obtain the reconstructed block of the first target CU. Thereafter, the reconstructed block of the first target CU is loop-filtered to obtain the reconstructed block of the first target CU after loop filtering. In this way, all the reconstructed blocks of the first target CU after loop filtering obtained by dividing the first image can constitute the first reconstructed image.

[0226] Loop filtering is performed on all reconstructed blocks of the CUs obtained by dividing the first image, that is, the image of the first area in the first reconstructed image and the image of the second image are both loop filtered. Optionally, after obtaining the first reconstructed image, the pixel values ​​of the pixels in the second area in the first reconstructed image can be set to a preset value, such as a dynamic range median (128).

[0227] Optionally, the reconstructed boundary pixels between the first area and the second area in the first reconstructed image are reconstructed pixels that have not been loop filtered. This can prevent the preset value filled in the second area in the first reconstructed image from contaminating the pixel values ​​in the second area. Optionally, when loop filtering is performed on the reconstructed pixels in the first area in the first reconstructed image, the reference reconstructed pixels do not include the reconstructed pixels in the second area. Optionally, the reconstructed pixels in the second area in the first image are reconstructed pixels that have not been loop filtered.

[0228] The boundary pixels can be determined based on the position indication information. Specifically, after obtaining the first reconstructed image, the pixel points in the second area of ​​the first reconstructed image can be determined based on the position indication information obtained above, and then the pixel values ​​of the pixel points in the second area of ​​the first reconstructed image are set to default values.

[0229] It should be noted that 503 and 504 are described by taking the example of skipping the encoding of non-first target CUs. When the encoding of non-first target CUs is not skipped, the encoding process of each first CU can refer to 503 and 504, and the embodiment of the present application will not be repeated here.

[0230] 506. Divide the second image to obtain multiple second CUs.

[0231] 507. Perform inter-frame prediction, intra-frame prediction or inter-layer prediction on multiple second CUs to obtain a predicted image of the second image.

[0232] The following is described by taking a second CU (hereinafter referred to as the second CU) among the multiple second CUs obtained by dividing the second image as an example, and reference may be made to S5051 to S5054:

[0233] 5071. For a second CU among the multiple CUs obtained by dividing the second image, determine whether the second CU belongs to the second region.

[0234] Exemplarily, for the second CU, it can be determined whether the second CU belongs to the second region. Specifically, the encoder can determine whether the second CU belongs to the second region according to the position information of the second CU and the position indication information obtained above.

[0235] When the second CU belongs to the second region, S5072 may be executed; when the second CU does not belong to the second region, S5073 may be executed.

[0236] 5072. Perform inter-frame prediction or intra-frame prediction on the second CU to determine a prediction block of the second CU.

[0237] Exemplarily, S5052 may refer to the description of the process of determining the prediction block of the first target CU in the above S5031, which will not be repeated here.

[0238] It should be noted that in S5052, the reference image for inter-frame prediction of the second CU is the second reconstructed image of other frames (that is, the reconstructed image of the second image of other frames); the reference block for intra-frame prediction of the second CU is the reconstructed block of the second image of the current frame.

[0239] 5073. Perform inter-layer prediction on the second CU to determine a prediction block of the second CU.

[0240] Exemplarily, the reconstructed image of the base layer, that is, the first reconstructed image, can be used as a reference image for inter-layer prediction to determine the prediction block of the second CU. Specifically, for the second CU, the reconstructed block at the same position as the second CU in the first reconstructed image can be used as the prediction block of the second CU. In this way, the MV of the second CU is (0,0), and the number of bits required to encode the MV of the second CU is relatively small, which can reduce the bit rate overhead of the encoding of the enhancement layer, thereby reducing the overall bit rate overhead. In addition, the amount of calculation of the motion search on the encoding end can also be reduced, thereby reducing the encoding complexity.

[0241] Figure 5C A prediction reference diagram is shown as an example.

[0242] Reference Figure 5C For example, assume that time k (k is a positive integer) is the current time, corresponding to the first image of the current frame and the second image of the current frame; time k-1 is the previous time, corresponding to the first reconstructed image of the previous frame and the second reconstructed image of the previous frame.

[0243] Reference Figure 5C For example, inter-frame prediction can be performed for the first CU in the first area of ​​the first image of the current frame, as shown in R01 and R02; wherein R01 indicates that the inter-frame prediction refers to the reconstructed block in the first area of ​​the first reconstructed image of the previous frame, and R02 indicates that the inter-frame prediction refers to the reconstructed block in the second area of ​​the first reconstructed image of the previous frame.

[0244] Reference Figure 5C For example, intra-frame prediction can be performed for the first CU in the first area of ​​the first image of the current frame, as shown in R03 and R04; wherein R03 indicates that the intra-frame prediction refers to the reconstructed block in the first area of ​​the first image of the current frame, and R04 indicates that the intra-frame prediction refers to the reconstructed block in the second area of ​​the first image of the current frame.

[0245] Reference Figure 5C For example, inter-frame prediction can be performed for the first CU in the second area in the first image of the current frame, as shown in R05 and R06; wherein R05 indicates that the inter-frame prediction refers to the reconstructed block in the first area in the first reconstructed image of the previous frame, and R06 indicates that the inter-frame prediction refers to the reconstructed block in the second area in the first reconstructed image of the previous frame.

[0246] Reference Figure 5C For example, intra-frame prediction can be performed for the first CU in the second area in the first image of the current frame, as shown in R07 and R08; wherein R07 indicates that the intra-frame prediction refers to the reconstructed block in the first area in the first image of the current frame, and R08 indicates that the intra-frame prediction refers to the reconstructed block in the second area in the first image of the current frame.

[0247] Reference Figure 5C For example, inter-layer prediction can be performed for the second CU in the first area of ​​the second image of the current frame, as shown in R09; wherein R09 indicates that the inter-layer prediction can refer to the reconstructed block of the first area in the first reconstructed image of the current frame (i.e., the reconstructed image of the base layer).

[0248] Reference Figure 5C , exemplarily, inter-frame prediction can be performed for the second CU in the second area in the second image of the current frame, as shown in R10 and R11; wherein R10 represents the inter-frame prediction referring to the reconstructed block in the first area in the second reconstructed image of the previous frame (the reconstructed image of the enhancement layer of the previous frame), and R11 represents the inter-frame prediction referring to the reconstructed block in the second area in the second reconstructed image of the previous frame.

[0249] Reference Figure 5C , exemplarily, intra-frame prediction can be performed for the second CU in the second area in the second image of the current frame, as shown in R12 and R13; wherein R12 indicates that the intra-frame prediction refers to the reconstructed block in the first area in the second image of the current frame, and R13 indicates that the intra-frame prediction refers to the reconstructed block in the second area in the second image of the current frame.

[0250] Optionally, the encoding of the second CU located in the first region can be skipped, that is, only the inter-frame prediction or intra-frame prediction of the EL is performed on each second target CU in the multiple second CUs. Exemplarily, the inter-frame prediction or intra-frame prediction of the enhancement layer can be performed on each second target CU in the multiple second CUs to obtain the prediction block of each second target CU, and the second target CU is the second CU located in the second region. The predicted image of the second image includes the prediction blocks of all second target CUs.

[0251] 508. Encode the residual between the second image and the predicted image of the second image to obtain an EL bit stream.

[0252] The residual between the second CU and the prediction block of the second CU may be encoded to obtain a bit stream of the second CU; wherein the bit streams of the plurality of second CUs obtained by dividing the second image constitute an EL bit stream.

[0253] Exemplarily, when the second CU does not belong to the second region (i.e., the second CU belongs to the first region), the residual between the second CU and the prediction block of the second CU can be forcibly set to 0. In this way, the number of bits required to encode the residual between the second CU and the prediction block of the second CU is relatively small, which can reduce the encoding bit rate overhead of the enhancement layer, thereby reducing the overall bit rate overhead.

[0254] It should be noted that the operations performed for encoding the residual between the prediction blocks of the second CU and the second CU include but are not limited to: transformation, quantization, and entropy coding, etc. For example, the residual between the prediction blocks of the second CU and the second CU may be transformed and quantized to obtain the residual coefficients of the second CU; then, the residual coefficients of the second CU may be entropy coded to obtain the bit stream of the second CU.

[0255] Exemplarily, a reconstruction operation may be performed during the encoding of the second image to obtain a second reconstructed image as a reference image for subsequent encoding of other second images of the frame. The second reconstructed image may also be referred to as a reconstructed image of the enhancement layer.

[0256] Reference Figure 5B (2) Exemplarily, when the second CU belongs to the second region, the residual coefficients of the second CU may be inversely quantized and inversely transformed to obtain the residual of the second CU; then, the residual of the second CU is added to the prediction block of the second CU to obtain the reconstructed block of the second CU. After that, the reconstructed block of the second CU may be loop filtered to obtain the reconstructed block of the second CU after loop filtering. When the second CU does not belong to the second region, the prediction block of the second CU may be used as the reconstructed block of the second CU, and there is no need to loop filter the reconstructed block of the second CU. In other words, the reconstructed blocks of the second CU located in the second region of all the second CUs obtained by dividing the second image are loop filtered, and the second CU located in the first region is not loop filtered, that is, the image of the first region in the second reconstructed image of the current frame is the same as the image of the first region in the first reconstructed image of the current frame.

[0257] The pixel value of the reconstructed block of the third target CU in the plurality of second CUs is equal to the pixel value of the reconstructed block in the first reconstructed image at the same position as the third target CU. The third target CU is a second CU located in the first region, the image of the second region in the second reconstructed image is an image that has undergone loop filtering, and the image of the first region in the second reconstructed image is an image that has not undergone loop filtering.

[0258] The reconstructed pixels at the boundary between the first region and the second region in the second image can always refer to the reconstructed pixels in the first region during loop filtering. It should be noted that, when loop filtering is performed on the reconstructed pixels in the second region in the second image, although the reconstructed pixels in the first region can be referred to, the reconstructed pixels in the first region cannot be modified.

[0259] Specifically, whether the second CU belongs to the second region may be determined according to the position indication information obtained above. For details, please refer to the above description, which will not be repeated here.

[0260] It should be noted that after encoding the second CU located in the first area in the second image, the motion information in the DPB will not be updated.

[0261] 509. Add the encoded position indication information to the BL bit stream and / or the EL bit stream.

[0262] Exemplarily, the encoded position indication information may be added to the front end or the end of the BL bit stream, and / or the encoded position indication information may be added to the end of the EL bit stream.

[0263] Exemplarily, after the encoded position indication information is added to the BL bitstream and / or the BL bitstream, the decoding end can constrain the loop filtering of the first area and the second area according to the position indication information parsed from the BL bitstream and / or the BL bitstream, and fuse the first reconstructed image and the second reconstructed image.

[0264] When the encoded position indication information is not added to the BL bitstream and / or the BL bitstream, the decoding end may derive the position information of the second region in other ways, which will be described in detail later.

[0265] It should be noted that the encoding end can also add encoding control parameters (such as prediction mode (which may include but is not limited to inter-frame prediction mode, intra-frame prediction mode and inter-layer prediction mode), division information (such as CU division method), transformation parameters and quantization parameters, etc.) to the bit stream of the original image, so that the decoding end can use the corresponding control parameters for decoding control and decode the reconstructed image of the original image.

[0266] Please refer to Fig. 6A , Fig. 6A A process diagram of another decoding method provided in an embodiment of the present application. Fig. 6A The decoding process and Figure 5A The encoding process corresponds to .

[0267] 601. Receive a bitstream, where the bitstream includes a BL bitstream and an EL bitstream, the base layer bitstream includes multiple first coding blocks, the enhancement layer bitstream includes multiple second coding blocks, the first coding block is a coding block of a first CU, and the second coding block is a coding block of a second CU.

[0268] The bitstream also includes position indication information.

[0269] Exemplarily, after receiving the bit stream, the decoder may parse the bit stream, determine each frame bit stream according to the frame identifier and layer identifier in the bit stream, and determine the BL bit stream and EL bit stream in each frame bit stream.

[0270] For example, when Figure 5AIn the embodiment, when the encoding end adds the encoded position indication information to the BL bit stream and / or the EL bit stream, the decoding end may also parse the position indication information from the received bit stream.

[0271] Exemplarily, the decoding end may also parse out control parameters (such as prediction mode (inter-frame prediction or intra-frame prediction), partition information (such as CU partitioning method), transform parameters and quantization parameters, etc.) from the received bit stream.

[0272] The following describes decoding of the BL bit stream and the EL bit stream in a frame bit stream.

[0273] Corresponding to the encoding end, when only the first target CU in the first image is encoded, the multiple first encoding blocks are encoding blocks of the first target CU in the first image, and the first target CU is the first CU located in the first area.

[0274] 602. Decode multiple first coding blocks to obtain a residual between a first image and a predicted image of the first image.

[0275] The prediction mode of the first coding block includes: inter-frame prediction and intra-frame prediction. When inter-frame prediction is performed on the first coding block, if the reference pixel of the bit stream of the first coding block is located in the second area of ​​the reference image, the pixel value of the reference pixel of the first coding block adopts a preset pixel value; if the reference motion information of the first coding block comes from the second area, the reference motion information is set to be invalid.

[0276] 603. Determine a reference image of the BL bit stream according to the reference frame identifier and the prediction mode in the BL bit stream.

[0277] When intra-frame prediction is performed on the coding block of the first target CU in the plurality of first coding blocks, the reference pixels of the coding block of the first target CU do not include the reconstructed pixels within the second region, and the first target CU is the first CU located within the second region. When inter-frame prediction is performed on the coding block of the first target CU, the reference motion information of the coding block of the first target CU does not include the motion information from the second region.

[0278] 604 . Predict multiple first coding blocks according to the reference image of the BL bit stream to obtain a predicted image of the first image.

[0279] Corresponding to the encoding end, when only the first target CU in the first image is encoded, the plurality of first encoding blocks are encoding blocks of the first target CU in the first image, and the first target CU is the first CU located in the first region. The predicted image of the first image includes the prediction blocks of all the first target CUs. Optionally, the residual between the prediction block of the first target CU and the first CU is 0.

[0280] The first coding block is predicted according to the reference image of the first coding block to determine the prediction block of the first coding block. Exemplarily, when the decoder determines that the preset mode of the first coding block is the inter-frame prediction mode according to the prediction mode parsed from the BL bit stream, the reference image of the first coding block can be determined from the first reconstructed image of other frames according to the reference frame identifier parsed from the BL bit stream; at this time, the decoder can also parse the MV of the first coding block from the BL bit stream. Then, according to the MV of the first coding block, the prediction block of the first coding block is determined from the reference image of the first coding block.

[0281] Exemplarily, when the decoder determines that the prediction mode of the first coding block is an intra-frame prediction mode based on the prediction mode parsed from the bit stream, the prediction block of the first coding block can be determined from the reconstructed pixels spatially adjacent to the first coding block based on the angle information parsed from the BL bit stream (or other information for determining the reference block or a pre-agreed intra-frame prediction reference rule).

[0282] 605. Add the predicted image of the first image to the residual decoded from the BL bit stream to obtain a first reconstructed image.

[0283] Figure 6B FIG. 4 is a schematic diagram of an exemplary decoding process.

[0284] The first reconstructed image is a reference image of a base layer bit stream and / or a reference image of an enhancement layer bit stream of a first image of another frame, and reconstructed boundary pixels between the first area and the second area in the first reconstructed image are reconstructed pixels that have not been loop filtered.

[0285] The reconstructed pixels in the second area of ​​the first reconstructed image are preset values. When loop filtering is performed on the reconstructed pixels in the first area of ​​the first reconstructed image, the referenced reconstructed pixels do not include the reconstructed pixels in the second area.

[0286] The decoder adds the residual of the first coding block parsed from the BL bitstream and the prediction block of the first coding block to obtain a reconstructed block of the first coding block, wherein the reconstructed blocks of the plurality of first coding blocks constitute a first reconstructed image.

[0287] Exemplarily, the decoder may also parse the residual of the first coding block from the BL bitstream; then, the residual of the first coding block and the prediction block of the first coding block may be added to obtain a reconstructed block of the first coding block; then, the reconstructed block of the first coding block may be loop filtered, and the details may refer to the above Figure 5A The loop filtering process of the reconstructed block of the first coding block in the embodiment is not described in detail here.

[0288] It should be noted that in Fig. 6AIn the embodiment, the second area in the first reconstructed image can be determined according to the position indication information parsed from the bit stream by the decoder; and then the pixel values ​​of the pixels in the second area in the first reconstructed image are set to a preset value, such as the median value of the dynamic range (128). By further agreeing on the reconstructed pixel filling value in the second area of ​​the BL layer at the codec end, it is only necessary to perform one decoding to decode the content of the EL layer. The image in the first area can be obtained by decoding the BL layer, and the image in the second area can be obtained by decoding the EL layer. The purpose of one-time encoding and one-time decoding is achieved, and the implementation cost is further reduced.

[0289] Optionally, the reconstructed boundary pixels between the first region and the second region in the first reconstructed image are reconstructed pixels that have not been loop filtered. This can prevent the preset value filled in the second region in the first reconstructed image from contaminating the pixel value in the second region. Optionally, when loop filtering is performed on the reconstructed pixels in the first region in the first reconstructed image, the reference reconstructed pixels do not include the reconstructed pixels in the second region.

[0290] The boundary pixels can be determined based on the position indication information. Specifically, after obtaining the first reconstructed image, the pixel points in the second area of ​​the first reconstructed image can be determined based on the position indication information obtained above, and then the pixel values ​​of the pixel points in the second area of ​​the first reconstructed image are set to default values.

[0291] 606. Decode multiple second coding blocks to obtain a residual between the second image and the predicted image of the second image.

[0292] The multiple second coding blocks are coding blocks of a second target CU in the second image, and the second target CU is a second CU located in the second region.

[0293] 607. Determine a reference image of the enhancement layer bit stream according to the reference frame identifier and the prediction mode in the enhancement layer bit stream.

[0294] 608. Predict multiple second coding blocks according to the reference image of the enhancement layer bit stream to obtain a predicted image of the second image.

[0295] The predicted image of the second image includes all the prediction blocks of the second target CU.

[0296] 609. Add the predicted image of the second image to the residual decoded from the enhancement layer bit stream to obtain a second reconstructed image.

[0297] The pixel value of the reconstructed block of the third target CU in the plurality of second CUs is equal to the pixel value of the reconstructed block in the first reconstructed image at the same position as the third target CU. The third target CU is a second CU located in the first region, the image of the second region in the second reconstructed image is an image that has undergone loop filtering, and the image of the first region in the second reconstructed image is an image that has not undergone loop filtering.

[0298] The second reconstructed image is a reconstructed image of the second image, the second reconstructed image is a reference image of the enhancement layer bitstream of the second image of other frames, and the pixel value of the reconstructed block of the coding block of the third target CU in the plurality of second coding blocks is equal to the pixel value of the reconstructed block in the first reconstructed image at the same position as the third target CU. The image of the second area in the second reconstructed image is an image that has been loop filtered, and the image of the first area in the second reconstructed image is an image that has not been loop filtered.

[0299] The second image is the original image, and in this case, the second reconstructed image can be used as the reconstructed image of the original image. The second image includes the image of the second area in the original image and does not include the image of the first area in the original image, and in this case, the second reconstructed image can be used as the reconstructed image of the original image.

[0300] Part of the video content that is not visible in the second area can be seen through the first reconstructed image, and the complete video can be viewed through the second reconstructed image. Fig. 7A FIG. 1 is a schematic diagram of a compression process. Fig. 7A In the example, the second image is the original image. In this case, the second reconstructed image can be used as the reconstructed image of the original image. Fig. 7A The encoding and decoding process can be referred to Figure 5A and Fig. 6A Description in (or refer to Fig. 7A and the description of Figure 8), which will not be repeated here.

[0301] Figure 7B FIG. 1 is a schematic diagram of a compression process. Figure 7B In the example, the second image includes the image of the second region in the original image and does not include the image of the first region in the original image. In this case, the second reconstructed image can be used as the reconstructed image of the original image; Figure 7B The encoding and decoding process can be referred to Figure 5A and Fig. 6A The description in will not be repeated here.

[0302] The sequence of the methods provided in the embodiments of the present application can be adjusted appropriately, and the process can be increased or decreased accordingly according to the situation. Any technician familiar with the technical field can easily think of a method of change within the technical scope disclosed in this application, which should be included in the protection scope of this application, and the embodiments of the present application do not limit this.

[0303] The above mainly introduces the encoding and decoding method provided by the embodiment of the present application from the perspective of the device. It is understandable that in order to realize the above functions, the device includes a hardware structure and / or software module corresponding to each function. Those skilled in the art should easily realize that, in combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0304] The embodiment of the present application can divide the functional modules of the device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0305] Fig. 8A A block diagram of a hierarchical coding device provided in an embodiment of the present application, in the case of dividing each functional module according to each function, the hierarchical coding device 800 may include an acquisition module 801, a division module 802, and a prediction coding module 803. Exemplarily, the coding device may be an encoder, or a chip thereof, or other combined devices, components, etc. having the functions of the above coding device. The functions of each module of the hierarchical coding device are as follows:

[0306] An acquisition module 801 is used to acquire a first image and a second image, wherein the first image includes an image of a first area in an original image and does not include an image of a second area in the original image, and the second image includes an image of the second area in the original image. The size of the first image and the size of the second image are the same as the size of the original image, and the image of the second area in the original image includes user information;

[0307] A division module 802 is used to divide the first image to obtain multiple first CUs, and any first CU among the multiple first CUs is located in the first area or in the second area;

[0308] The prediction coding module 803 is used to perform prediction coding of a base layer on multiple first CUs to obtain a base layer bit stream, where the prediction mode of the first CU includes: inter-frame prediction and intra-frame prediction; when inter-frame prediction is performed on the first CU, if the reference pixel of the first CU is located in the second area of ​​the reference image, the pixel value of the reference pixel of the first CU adopts a preset pixel value; if the reference motion information of the first CU comes from the second area, the reference motion information is set to be invalid;

[0309] The division module 802 is further used to divide the second image to obtain multiple second CUs, and any second CU of the multiple second CUs is located in the first area or in the second area;

[0310] The prediction coding module 803 is further used to perform prediction coding of enhancement layers on the plurality of second CUs to obtain enhancement layer bit streams, and when the second CU is located within the first region, the prediction mode of the second CU includes: inter-layer prediction, and the reference block of the second CU is a reconstructed block in the first reconstructed image at the same position as the second CU, and the first reconstructed image is a reconstructed image of the first image; when the second CU is located within the second region, the prediction mode of the second CU includes: inter-frame prediction and intra-frame prediction;

[0311] Among them, the base layer bit stream and the enhancement layer bit stream are interwoven into one bit stream.

[0312] In combination with the above scheme, the prediction coding module 803 is specifically used to: perform inter-frame prediction or intra-frame prediction on multiple first CUs to obtain a predicted image of the first image; encode the residual between the first image and the predicted image of the first image to obtain a basic layer bit stream.

[0313] In combination with the above scheme, the prediction coding module 803 is specifically used to: perform inter-frame prediction or intra-frame prediction of the base layer on each first target CU among multiple first CUs to obtain a prediction block of each first target CU, where the first target CU is the first CU located in the first area; wherein the predicted image of the first image includes the prediction blocks of all first target CUs.

[0314] In conjunction with the above solutions, please refer to Figure 8B , Figure 8B A block diagram of another hierarchical coding device provided in an embodiment of the present application, in Fig. 8A On the basis of Figure 8B It also includes: a reconstruction module 804, which is used to generate a reconstructed block of each first CU in multiple first CUs to obtain a first reconstructed image; wherein the first reconstructed image is a reference image of the first images of other frames and / or a reference image of the second image, and the reconstructed boundary pixels between the first area and the second area in the first reconstructed image are reconstructed pixels that have not undergone loop filtering.

[0315] In combination with the above solution, the reconstructed pixels of the second area in the first reconstructed image are preset values.

[0316] In combination with the above solution, when loop filtering is performed on the reconstructed pixels in the first area of ​​the first reconstructed image, the reconstructed pixels used as reference do not include the reconstructed pixels in the second area.

[0317] In combination with the above scheme, when intra-frame prediction is performed on the first target CU among multiple first CUs, the reference pixels of the first target CU do not include the reconstructed pixels within the second area, and the first target CU is the first CU located within the first area; when inter-frame prediction is performed on the first target CU, the reference motion information of the first target CU does not include motion information from the second area.

[0318] In combination with the above scheme, the prediction coding module 803 is specifically used to: perform inter-frame prediction, intra-frame prediction or inter-layer prediction on multiple second CUs to obtain a predicted image of the second image; encode the residual between the second image and the predicted image of the second image to obtain an enhanced layer bit stream.

[0319] In combination with the above scheme, the prediction coding module 803 is specifically used to: perform inter-frame prediction or intra-frame prediction of the enhancement layer on each second target CU in multiple second CUs to obtain a prediction block of each second target CU, where the second target CU is a second CU located in the second area; wherein the predicted image of the second image includes the prediction blocks of all second target CUs.

[0320] In combination with the above scheme, the reconstruction module 804 is also used to generate a reconstructed block of each second CU in multiple second CUs to obtain a second reconstructed image; wherein the second reconstructed image is a reconstructed image of the second image, the second reconstructed image is a reference image of the second images of other frames, the pixel value of the reconstructed block of the third target CU in the multiple second CUs is equal to the pixel value of the reconstructed block with the same position as the third target CU in the first reconstructed image; the third target CU is the second CU located in the first area, the image of the second area in the second reconstructed image is an image that has undergone loop filtering, and the image of the first area in the second reconstructed image is an image that has not undergone loop filtering.

[0321] In combination with the above solution, the residual between the prediction block of the third target CU and the first CU is 0.

[0322] In combination with the above solution, the second image also includes the image of the first area in the original image; or, the second image does not include the image of the first area in the original image.

[0323] In combination with the above scheme, the acquisition module 801 is further used to obtain position indication information, where the position indication information is used to indicate the position of the second area; the prediction coding module 803 is further used to add the encoded position indication information to the base layer bit stream and / or the enhancement layer bit stream.

[0324] In combination with the above scheme, the first area in the original image is the area in the original image excluding the second area; the second image is generated by replacing the pixel values ​​of the pixel points in the first area in the original image; the first image is generated by replacing the pixel values ​​of the pixel points in the second area in the original image; wherein the replacement method includes at least one of the following: solid color filling, mosaic or Gaussian blur.

[0325] In combination with the above solution, the base layer bit stream and the enhancement layer bit stream include a reference frame identifier and a prediction mode.

[0326] Fig. 9 A block diagram of a hierarchical decoding device provided in an embodiment of the present application, in the case where each functional module is divided according to each function, the hierarchical decoding device 900 may include a transceiver module 901 and a predictive decoding module 902. Exemplarily, the decoding device may be a decoder, or a chip thereof, or other combined devices, components, etc. having the functions of the above-mentioned decoding device. The functions of each module of the decoding device are as follows:

[0327] The transceiver module 901 is used to receive a bit stream, where the bit stream includes a base layer bit stream and an enhancement layer bit stream, where the base layer bit stream includes multiple first coding blocks, and the enhancement layer bit stream includes multiple second coding blocks, where the first coding block is a coding block of a first CU, and the second coding block is a coding block of a second CU;

[0328] The prediction decoding module 902 is used to perform prediction decoding of a base layer on a plurality of first coding blocks to obtain a first reconstructed image, wherein the first reconstructed image is a reconstructed image of a first image, the first image includes an image of a first area in an original image and does not include an image of a second area in the original image, the image of the second area in the original image includes user information, and the prediction mode of the first coding block includes: inter-frame prediction and intra-frame prediction; when inter-frame prediction is performed on the first coding block, if a reference pixel of a bit stream of the first coding block is located in the second area in the reference image, a pixel value of the reference pixel of the first coding block adopts a preset pixel value; if reference motion information of the first coding block comes from the second area, the reference motion information is set to be invalid;

[0329] The prediction decoding module 902 is further used to perform prediction decoding of the enhancement layer on the plurality of second coding blocks to obtain a second reconstructed image, wherein the second reconstructed image is a reconstructed image of the second image, the second image includes an image of the second area in the original image, the size of the first image and the size of the second image are the same as the size of the original image, and when the second CU corresponding to the second coding block is located within the first area, the prediction mode of the second coding block includes: inter-layer prediction, and the reference block of the second coding block is a reconstructed block in the first reconstructed image with the same position as the second CU corresponding to the second coding block; when the second CU corresponding to the second coding block is located within the second area, the prediction mode of the second coding block includes: inter-frame prediction and intra-frame prediction;

[0330] The first reconstructed image and the second reconstructed image are used to determine the reconstructed image of the original image.

[0331] In combination with the above scheme, the base layer bit stream includes a reference frame identifier and a prediction mode, and the prediction decoding module 902 is specifically used to: decode multiple first coding blocks to obtain a residual between a first image and a predicted image of the first image; determine the reference image of the base layer bit stream according to the reference frame identifier and the prediction mode in the base layer bit stream; predict multiple first coding blocks according to the reference image of the base layer bit stream to obtain a predicted image of the first image; add the predicted image of the first image to the residual decoded from the base layer bit stream to obtain a first reconstructed image.

[0332] In combination with the above scheme, the multiple first coding blocks are coding blocks of the first target CU in the first image, and the first target CU is the first CU located in the first area; wherein the predicted image of the first image includes the prediction blocks of all the first target CUs.

[0333] In combination with the above solution, the residual between the prediction block of the first target CU and the first CU is zero.

[0334] In combination with the above scheme, the first reconstructed image is a reference image of the basic layer bit stream and / or a reference image of the enhanced layer bit stream of the first image of other frames, and the reconstructed boundary pixels between the first area and the second area in the first reconstructed image are reconstructed pixels that have not undergone loop filtering.

[0335] In combination with the above solution, the reconstructed pixels of the second area in the first reconstructed image are preset values.

[0336] In combination with the above solution, when loop filtering is performed on the reconstructed pixels in the first area of ​​the first reconstructed image, the reconstructed pixels used as reference do not include the reconstructed pixels in the second area.

[0337] In combination with the above scheme, when intra-frame prediction is performed on the coding block of the first target CU among multiple first coding blocks, the reference pixels of the coding block of the first target CU do not include the reconstructed pixels within the second area, and the first target CU is the first CU located within the second area; when inter-frame prediction is performed on the coding block of the first target CU, the reference motion information of the coding block of the first target CU does not include motion information from the second area.

[0338] In combination with the above scheme, the enhancement layer bit stream includes a reference frame identifier and a prediction mode, and the prediction decoding module 902 is specifically used to: decode multiple second coding blocks to obtain the residual between the second image and the predicted image of the second image; determine the reference image of the enhancement layer bit stream according to the reference frame identifier and the prediction mode in the enhancement layer bit stream; predict multiple second coding blocks according to the reference image of the enhancement layer bit stream to obtain the predicted image of the second image; add the predicted image of the second image to the residual decoded from the enhancement layer bit stream to obtain a second reconstructed image.

[0339] In combination with the above scheme, the multiple second coding blocks are coding blocks of the second target CU in the second image, and the second target CU is the second CU located in the second area; wherein the predicted image of the second image includes the prediction blocks of all the second target CUs.

[0340] In combination with the above scheme, the second reconstructed image is a reconstructed image of the second image, the second reconstructed image is a reference image of the enhancement layer bit stream of the second image of other frames, the pixel value of the reconstructed block of the coding block of the third target CU in the multiple second coding blocks is equal to the pixel value of the reconstructed block with the same position as the third target CU in the first reconstructed image; the third target CU is the second CU located in the first area, the image of the second area in the second reconstructed image is an image that has undergone loop filtering, and the image of the first area in the second reconstructed image is an image that has not undergone loop filtering.

[0341] In combination with the above solution, the second image also includes the image of the first area in the original image; or, the second image does not include the image of the first area in the original image.

[0342] In combination with the above solution, the bit stream also includes encoded position indication information, where the position indication information is used to indicate the position of the second area; the prediction decoding module 902 is further used to determine the second area in the first reconstructed image according to the position indication information.

[0343] In combination with the above solution, the prediction decoding module 902 is further configured to use the second reconstructed image as the reconstructed image of the original image.

[0344] Fig.101 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 1000 may be an encoder or a decoder, or a chip or a functional module in an encoder or a decoder. Fig.10 As shown, the electronic device 1000 includes a processor 1001 , a transceiver 1002 and a communication line 1003 .

[0345] The processor 1001 is used to execute any step in the method embodiment shown above, and when executing processes such as acquiring the first image and the second image, the transceiver 1002 and the communication line 1003 can be selectively called to complete the corresponding operations.

[0346] Furthermore, the electronic device 1000 may further include a memory 1004. The processor 1001, the memory 1004 and the transceiver 1002 may be connected via a communication line 1003.

[0347] The transceiver 1002 is used to communicate with other devices or other communication networks, and other communication networks may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The transceiver 1002 may be a module, a circuit, a transceiver or any device capable of achieving communication.

[0348] The transceiver 1002 is mainly used for receiving and sending bit streams, etc., and may include a transmitter and a receiver, which respectively send and receive bit streams, etc.; operations other than receiving and sending bit streams, etc. are implemented by the processor, such as predictive coding of the base layer of multiple first CUs, etc.

[0349] The communication line 1003 is used to transmit information between the components included in the electronic device 1000.

[0350] In one design, the processor may be considered as the logic circuit and the transceiver may be considered as the interface circuit.

[0351] The memory 1004 is used to store instructions, where the instructions may be computer programs.

[0352] It should be noted that the memory 1004 can exist independently of the processor 1001, or can be integrated with the processor 1001. The memory 1004 can be used to store instructions or program codes or some data, etc. The memory 1004 can be located inside the electronic device 1000, or outside the electronic device 1000, without limitation. The processor 1001 is used to execute the instructions stored in the memory 1004 to implement the method provided in the above embodiment of the present application.

[0353] In one example, the processor 1001 may include one or more processors, such as Fig.10 Processor 0 and processor 1 in.

[0354] As an optional implementation, the electronic device 1000 includes multiple processors, for example, Fig.10 In addition to the processor 1001, a processor 1007 may also be included.

[0355] As an optional implementation, the electronic device 1000 further includes an output device 1005 and an input device 1006. Exemplarily, the input device 1006 is a keyboard, a mouse, a microphone, a joystick, and the output device 1005 is a display screen, a speaker, and the like.

[0356] It should be noted that the electronic device 1000 may be a chip network or a Fig.10 Devices with similar structures in the chip network. Among them, the chip network can be composed of chips, or it can include chips and other discrete devices. The actions, terms, etc. involved in the various embodiments of this application can refer to each other without limitation. The message name or parameter name in the message exchanged between the various devices in the embodiment of this application is only an example, and other names can also be used in the specific implementation without limitation. In addition, Fig.10 The components shown in the figure do not constitute a limitation on the electronic device 1000. Fig.10 In addition to the components shown, the electronic device 1000 may include Fig.10 More or fewer components may be shown, or certain components may be combined, or the components may be arranged differently.

[0357] The processor and transceiver described in the present application can be implemented in an integrated circuit (IC), an analog IC, a radio frequency integrated circuit, a mixed signal IC, an application specific integrated circuit (ASIC), a printed circuit board (PCB), an electronic device, etc. The processor and transceiver can also be manufactured using various IC process technologies, such as complementary metal oxide semiconductor (CMOS), N-type metal oxide semiconductor (NMOS), P-type metal oxide semiconductor (positive channel metal oxide semiconductor, PMOS), bipolar junction transistor (BJT), bipolar CMOS (BiCMOS), silicon germanium (SiGe), gallium arsenide (GaAs), etc.

[0358] Fig.11 The following is a schematic diagram of the structure of a layered coding and decoding device provided in an embodiment of the present application. The layered coding and decoding device can be applied to the scenario shown in the above method embodiment. For the convenience of explanation, Fig.11 Only the main components of the layered coding and decoding device are shown, including a processor, a memory, a control circuit, and an input-output device. The processor is mainly used to process the communication protocol and the communication image, execute the software program, and process the data of the software program. The memory is mainly used to store the software program and data. The control circuit is mainly used for power supply and transmission of various electrical signals. The input-output device is mainly used to receive data input by the user and output data to the user.

[0359] When the layered coding and decoding device is an encoder or decoder, the control circuit can be a mainboard, the memory includes a hard disk, RAM, ROM and other media with storage functions, the processor can include a baseband processor and a central processing unit, the baseband processor is mainly used to process the communication protocol and communication data, the central processing unit is mainly used to control the entire layered coding and decoding device, execute software programs, and process the data of software programs, and the input and output devices include a display screen, a keyboard and a mouse, etc.; the control circuit can further include or connect a transceiver circuit or a transceiver, such as: a network cable interface, etc., for sending or receiving data or signals, such as data transmission and communication with other devices. Furthermore, it can also include an antenna for sending and receiving wireless signals, and for data / signal transmission with other devices.

[0360] According to the method provided in the embodiments of the present application, the present application also provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer executes any of the methods described in the embodiments of the present application.

[0361] The embodiment of the present application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be executed by a computer or a device with layered coding and decoding capabilities to control the relevant hardware to complete, and the computer program or the group of instructions can be stored in the above computer-readable storage medium. When the computer program or the group of instructions is executed, it may include the processes of the above method embodiments. The computer-readable storage medium may be an internal storage unit of the encoder or decoder of any of the above embodiments, such as a hard disk or memory of the encoder or decoder. The above computer-readable storage medium may also be an external storage device of the above encoder or decoder, such as a plug-in hard disk equipped on the above encoder or decoder, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. Further, the above computer-readable storage medium may also include both the internal storage unit of the above encoder or decoder and an external storage device. The above computer-readable storage medium is used to store the above computer program or instructions and other programs and data required by the above encoder or decoder. The above computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0362] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0363] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0364] In the several embodiments provided in the present application, it should be understood that the disclosed networks, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another network, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0365] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0366] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0367] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks or optical disks.

[0368] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A layered coding method, characterized in that: The method comprises: Acquire a first image and a second image, wherein the first image includes an image of a first area in an original image and does not include an image of a second area in the original image, and the second image includes an image of the second area in the original image, a size of the first image and a size of the second image are the same as a size of the original image, and the image of the second area in the original image includes user information; Dividing the first image to obtain a plurality of first coding units CU, wherein any first CU of the plurality of first CUs is located within the first region or within the second region; Performing prediction encoding of a base layer on a plurality of first CUs to obtain a base layer bitstream, wherein the prediction mode of the first CU includes: inter-frame prediction and intra-frame prediction; when performing inter-frame prediction on the first CU, if a reference pixel of the first CU is located in the second area in the reference image, a pixel value of the reference pixel of the first CU adopts a preset pixel value; if reference motion information of the first CU comes from the second area, the reference motion information is set to be invalid; Dividing the second image to obtain a plurality of second CUs, any second CU of the plurality of second CUs being located within the first region or within the second region; Performing prediction encoding of enhancement layers on the multiple second CUs to obtain an enhancement layer bitstream, when the second CU is located within the first region, the prediction mode of the second CU includes: inter-layer prediction, and the reference block of the second CU is a reconstructed block in the first reconstructed image at the same position as the second CU, and the first reconstructed image is a reconstructed image of the first image; when the second CU is located within the second region, the prediction mode of the second CU includes: inter-frame prediction and intra-frame prediction; The base layer bit stream and the enhancement layer bit stream are interleaved into one bit stream.

2. The method according to claim 1, characterized in that: The step of performing predictive coding on a base layer of the plurality of first CUs to obtain a base layer bit stream includes: Performing inter-frame prediction or intra-frame prediction on the multiple first CUs to obtain a predicted image of the first image; A residual between the first image and a predicted image of the first image is encoded to obtain the base layer bit stream.

3. The method according to claim 2, characterized in that The performing inter-frame prediction or intra-frame prediction on the multiple first CUs to obtain a predicted image of the first image includes: Performing inter-frame prediction or intra-frame prediction of the base layer on each first target CU among the multiple first CUs to obtain a prediction block of each first target CU, where the first target CU is a first CU located in the first region; The predicted image of the first image includes all the prediction blocks of the first target CU.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Generate a reconstructed block of each first CU in the multiple first CUs to obtain the first reconstructed image; The first reconstructed image is a reference image of the first image of other frames and / or a reference image of the second image, and the reconstructed boundary pixels between the first area and the second area in the first reconstructed image are reconstructed pixels that have not been loop filtered.

5. The method according to claim 4, characterized in that The reconstructed pixels of the second area in the first reconstructed image are preset values.

6. The method according to claim 4 or 5, characterized in that: When loop filtering is performed on the reconstructed pixels in the first area of ​​the first reconstructed image, the reconstructed pixels used as reference do not include the reconstructed pixels in the second area.

7. The method according to any one of claims 1 to 6, characterized in that When intra prediction is performed on a first target CU among the multiple first CUs, reference pixels of the first target CU do not include reconstructed pixels within the second area, and the first target CU is a first CU located within the first area; When inter-frame prediction is performed on the first target CU, the reference motion information of the first target CU does not include motion information from the second region.

8. The method according to any one of claims 1 to 7, characterized in that The performing prediction coding of the enhancement layers on the multiple second CUs to obtain an enhancement layer bit stream includes: Performing inter-frame prediction, intra-frame prediction or inter-layer prediction on the multiple second CUs to obtain a predicted image of the second image; A residual between the second image and a predicted image of the second image is encoded to obtain the enhancement layer bitstream.

9. The method according to claim 8, characterized in that The performing inter-frame prediction, intra-frame prediction or inter-layer prediction on the multiple second CUs to obtain a predicted image of the second image includes: Performing inter-frame prediction or intra-frame prediction of an enhancement layer on each second target CU in the multiple second CUs to obtain a prediction block of each second target CU, where the second target CU is a second CU located in the second region; The predicted image of the second image includes all the prediction blocks of the second target CU.

10. The method according to any one of claims 1 to 9, characterized in that The method further comprises: Generate a reconstructed block of each second CU in the multiple second CUs to obtain a second reconstructed image; The second reconstructed image is a reconstructed image of the second image, the second reconstructed image is a reference image of the second images of other frames, and the pixel value of the reconstructed block of the third target CU in the plurality of second CUs is equal to the pixel value of the reconstructed block in the first reconstructed image at the same position as the third target CU; The third target CU is a second CU located in the first area, the image of the second area in the second reconstructed image is an image that has undergone loop filtering, and the image of the first area in the second reconstructed image is an image that has not undergone loop filtering.

11. The method according to claim 10, characterized in that A residual difference between the prediction block of the third target CU and the first CU is 0.

12. The method according to any one of claims 1 to 11, characterized in that The second image also includes an image of the first area in the original image; or, The second image does not include an image of the first area in the original image.

13. The method according to any one of claims 1 to 12, characterized in that The method further comprises: Acquire location indication information, where the location indication information is used to indicate the location of the second area; The encoded position indication information is added to the base layer bit stream and / or the enhancement layer bit stream.

14. The method according to any one of claims 1 to 13, characterized in that The first area in the original image is an area in the original image excluding the second area; The second image is generated by replacing pixel values ​​of pixel points in the first area of ​​the original image; The first image is generated by replacing pixel values ​​of pixel points in a second area in the original image; The replacement method includes at least one of the following: solid color filling, mosaic or Gaussian blur.

15. The method according to any one of claims 1 to 14, characterized in that The base layer bitstream and the enhancement layer bitstream include reference frame identification and prediction mode.

16. A layered decoding method, characterized in that: The method comprises: Receive a bitstream, the bitstream comprising a base layer bitstream and an enhancement layer bitstream, the base layer bitstream comprising a plurality of first coding blocks, the enhancement layer bitstream comprising a plurality of second coding blocks, the first coding blocks being coding blocks of a first CU, and the second coding blocks being coding blocks of a second CU; Performing prediction decoding of a base layer on the multiple first coding blocks to obtain a first reconstructed image, wherein the first reconstructed image is a reconstructed image of a first image, the first image includes an image of a first area in an original image and does not include an image of a second area in the original image, the image of the second area in the original image includes user information, and the prediction mode of the first coding block includes: inter-frame prediction and intra-frame prediction; when inter-frame prediction is performed on the first coding block, if a reference pixel of a bit stream of the first coding block is located in the second area in the reference image, a pixel value of the reference pixel of the first coding block adopts a preset pixel value; if reference motion information of the first coding block comes from the second area, the reference motion information is set to be invalid; Performing prediction decoding of the enhancement layer on the multiple second coding blocks to obtain a second reconstructed image, wherein the second reconstructed image is a reconstructed image of the second image, the second image includes an image of the second area in the original image, the size of the first image and the size of the second image are the same as the size of the original image, and when the second CU corresponding to the second coding block is located within the first area, the prediction mode of the second coding block includes: inter-layer prediction, and the reference block of the second coding block is a reconstructed block in the first reconstructed image with the same position as the second CU corresponding to the second coding block; when the second CU corresponding to the second coding block is located within the second area, the prediction mode of the second coding block includes: inter-frame prediction and intra-frame prediction; The first reconstructed image and the second reconstructed image are used to determine the reconstructed image of the original image.

17. The method according to claim 16, characterized in that The base layer bit stream includes a reference frame identifier and a prediction mode, and the base layer prediction decoding is performed on the plurality of first coding blocks to obtain a first reconstructed image, including: Decoding the plurality of first coding blocks to obtain a residual between the first image and a predicted image of the first image; determining a reference picture of the base layer bitstream according to the reference frame identifier and the prediction mode in the base layer bitstream; Predicting the plurality of first coding blocks according to a reference image of the base layer bitstream to obtain a predicted image of the first image; The predicted image of the first image is added to the residual decoded from the base layer bit stream to obtain the first reconstructed image.

18. The method according to claim 17, characterized in that The multiple first coding blocks are coding blocks of a first target CU in the first image, and the first target CU is a first CU located in the first area; The predicted image of the first image includes all the prediction blocks of the first target CU.

19. The method according to claim 18, characterized in that A residual between the prediction block of the first target CU and the first CU is 0.

20. The method according to any one of claims 16 to 19, characterized in that The first reconstructed image is a reference image of a base layer bit stream of a first image of another frame and / or a reference image of an enhancement layer bit stream, and the reconstructed boundary pixels between the first area and the second area in the first reconstructed image are reconstructed pixels that have not been loop filtered.

21. The method according to claim 20, characterized in that The reconstructed pixels of the second area in the first reconstructed image are preset values.

22. The method according to claim 20 or 21, characterized in that When loop filtering is performed on the reconstructed pixels in the first area of ​​the first reconstructed image, the reconstructed pixels used as reference do not include the reconstructed pixels in the second area.

23. The method according to any one of claims 16 to 22, characterized in that When intra prediction is performed on a coding block of a first target CU among the multiple first coding blocks, reference pixels of the coding block of the first target CU do not include reconstructed pixels within the second area, and the first target CU is a first CU located within the second area; When inter-frame prediction is performed on the coding block of the first target CU, the reference motion information of the coding block of the first target CU does not include motion information from the second region.

24. The method according to any one of claims 16 to 23, characterized in that The enhancement layer bitstream includes a reference frame identifier and a prediction mode, and performing prediction decoding of the enhancement layer on the plurality of second coding blocks to obtain a second reconstructed image includes: Decoding the plurality of second coding blocks to obtain a residual between the second image and a predicted image of the second image; determining a reference picture of the enhancement layer bitstream according to the reference frame identifier and the prediction mode in the enhancement layer bitstream; Predicting the plurality of second coding blocks according to a reference image of the enhancement layer bitstream to obtain a predicted image of the second image; The predicted image of the second image is added to the residual decoded from the enhancement layer bit stream to obtain the second reconstructed image.

25. The method according to claim 24, characterized in that The plurality of second coding blocks are coding blocks of a second target CU in the second image, and the second target CU is a second CU located in the second area; The predicted image of the second image includes all the prediction blocks of the second target CU.

26. The method according to any one of claims 16 to 25, characterized in that The second reconstructed image is a reconstructed image of the second image, the second reconstructed image is a reference image of an enhancement layer bit stream of the second image of another frame, and the pixel value of the reconstructed block of the coding block of the third target CU in the plurality of second coding blocks is equal to the pixel value of the reconstructed block at the same position as the third target CU in the first reconstructed image; The third target CU is a second CU located in the first area, the image of the second area in the second reconstructed image is an image that has undergone loop filtering, and the image of the first area in the second reconstructed image is an image that has not undergone loop filtering.

27. The method according to any one of claims 16 to 26, characterized in that The second image also includes an image of the first area in the original image; or, The second image does not include an image of the first area in the original image.

28. The method according to any one of claims 16 to 27, characterized in that The bit stream further includes encoded position indication information, where the position indication information is used to indicate the position of the second area; and the method further includes: A second area in the first reconstructed image is determined according to the position indication information.

29. The method according to any one of claims 16 to 28, characterized in that The method further comprises: The second reconstructed image is used as the reconstructed image of the original image.

30. A layered coding device, characterized in that: The device comprises: one or more processors; a memory for storing one or more computer programs or instructions; When one or more computer programs or instructions are executed by one or more processors, the one or more processors implement the method according to any one of claims 1 to 15.

31. A layered decoding device, characterized in that: The device comprises: one or more processors; a memory for storing one or more computer programs or instructions; When one or more computer programs or instructions are executed by one or more processors, the one or more processors implement the method according to any one of claims 16 to 29.

32. A chip, characterized in that: include: A processing circuit and an interface circuit; wherein the interface circuit is used to couple with the memory and provide a communication interface for the processing circuit to access the memory; The processing circuit is used to execute program instructions in the memory to implement the method according to any one of claims 1 to 29.

Images