Residual encoding, decoding method and device, storage medium and electronic device

Abstract

The present invention provides a residual coding and decoding method and device, a storage medium and an electronic device. In the residual coding method, the residual of the reference block of the block to be coded is determined; based on the residual of the reference block of the block to be coded, the residual of the block to be coded is predicted to obtain the predicted residual of the block to be coded; the residual difference between the predicted residual of the block to be coded and the actual residual of the block to be coded is encoded into a bit stream. Through the present invention, the problem of how to reduce the bit rate of the coded residual in the related art is solved, thereby achieving the effect of saving the bit rate.

Description

Technical Field

[0001] The present invention relates to the field of communications, and in particular to a residual encoding and decoding method and device, a storage medium and an electronic device. Background Art

[0002] In recent years, with the rapid development of network communication and multimedia technology, video content has begun to be presented to viewers in the form of high resolution and ultra-high resolution. Compared with standard resolution video, the higher the video resolution, the better the visual experience of people watching video content. At the same time, high-resolution video also puts higher requirements on video coding technology. In order to meet this challenge, the Joint Collaborative Team on Video Coding (JCT-VC) has developed a new generation of high-efficiency video coding standard (High Efficiency Video Coding, HEVC for short). Compared with the previous generation video coding standard H.264 / MPEG-4 AVC, HEVC improves compression efficiency by 50% and maintains the same original visual quality. On September 1, 2015, Google announced the establishment of the Alliance of Open Media (AOM) with Amazon, Cisco, Intel, Microsoft, Firefox, and Netflix. AOM developed a new generation of video coding format AV1. In December 2017, China's Audio Video Coding Standard (AVS) proposed a new generation of AVS3 video coding. The Joint Video Exploration Team (JVET) determined the name of the latest generation of video coding standards as Versatile Video Coding (VVC) at a conference in San Diego, USA on April 10, 2018. The main goal is to improve the existing HEVC and provide higher compression performance. At the same time, it will optimize for emerging applications (360° panoramic video and High Dynamic Range Imaging (HDR or HDRI)). VVC is expected to be standardized before 2020, and the current proposed solution has improved by more than 40% compared to HEVC.

[0003] Inter-frame prediction is the core component of the video coding standards HEVC / AV1 / AVS. It uses the correlation of the video time domain and uses the pixels of the adjacent coded images in the time domain to predict the pixels of the current image, so as to effectively remove the temporal redundancy of the video. At present, the mainstream video coding standards all use block-based motion compensation technology. The principle is to find a best matching block in the previously coded image through motion estimation for each pixel block of the current image. The image used for prediction is called the reference image, the displacement from the reference block to the current pixel block is called the motion vector (MV for short), and the difference between the original pixel value of the current block and the pixel value of the predicted block after motion compensation of the reference block is called the residual (also called residual, residual value, residual block). Inter-frame prediction only needs to write the optimal MV, reference frame index and residual value of the coded block into the bitstream after encoding and transmit it to the decoder. The decoder finds the corresponding reference block in the reference frame according to the optimal MV and reference frame index, and then adds the decoded residual value to restore the original pixel value of the decoded block. The main bit rate consumption of inter-frame prediction is the encoding of the residual block. Traditional inter-frame prediction directly encodes the actual residual obtained after prediction. However, in most complex motion situations, the value of the residual block is very large, which will result in a very high bit rate for encoding the residual.

[0004] Therefore, how to reduce the bit rate of the coding residual to achieve the effect of saving bit rate is a problem that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present invention provide a residual coding and decoding method and device, a storage medium and an electronic device to at least solve the problem of how to reduce the bit rate of the coding residual in the related art.

[0006] According to one embodiment of the present invention, a residual coding method is provided, comprising: determining a residual of a reference block of a block to be coded, wherein the reference block of the block to be coded comprises: at least two first reference blocks of the block to be coded in the time domain; or, at least two second reference blocks of the block to be coded in the spatial domain; or, at least one first reference block of the block to be coded in the time domain and at least one second reference block of the block to be coded in the spatial domain; predicting the residual of the block to be coded based on the residual of the reference block of the block to be coded to obtain a predicted residual of the block to be coded; and encoding a residual difference value between the predicted residual of the block to be coded and an actual residual of the block to be coded into a bitstream.

[0007] In at least one exemplary embodiment, when the image frame where the block to be encoded is located is a P frame, the first reference block of the block to be encoded in the time domain includes: the optimal prediction unit block PU of the block to be encoded in the forward reference frame, recorded as the first optimal PU, and / or, the optimal PU of the block to be encoded in the previous frame of the forward reference frame, recorded as the second optimal PU; or, when the image frame where the block to be encoded is located is a B frame, the first reference block of the block to be encoded in the time domain includes: the optimal PU of the block to be encoded in the forward reference frame, recorded as the third optimal PU, and / or, the optimal PU of the block to be encoded in the backward reference frame, recorded as the fourth optimal PU.

[0008] In at least one exemplary embodiment, when the image frame in which the block to be encoded is located is a P frame, the first reference block of the block to be encoded in the time domain is determined in the following manner: determining the first optimal PU of the block to be encoded in the forward reference frame by motion estimation, and determining the motion vector MV of the block to be encoded relative to the first optimal PU; and determining the second optimal PU in a frame previous to the forward reference frame by motion estimation according to the position of the first optimal PU and the MV.

[0009] In at least one exemplary embodiment, the second reference block of the block to be encoded in the spatial domain is located in the image frame where the block to be encoded is located, and is adjacent to the block to be encoded in the spatial domain.

[0010] In at least one exemplary embodiment, the second reference block of the block to be encoded in the spatial domain includes: a left block and / or an upper block adjacent to the block to be encoded in the image frame where the block to be encoded is located.

[0011] In at least one exemplary embodiment, the reference block of the block to be encoded comprises:

[0012] Two first reference blocks whose corresponding residuals of the block to be encoded in the time domain are not all zero, and two second reference blocks whose corresponding residuals of the block to be encoded in the spatial domain are not all zero; or

[0013] A first reference block whose corresponding residual of the block to be encoded in the time domain is non-all zero, and a second reference block whose corresponding residual of the block to be encoded in the spatial domain is non-all zero; or

[0014] Two first reference blocks whose corresponding residuals of the block to be encoded in the time domain are non-all zero, and one second reference block whose corresponding residuals of the block to be encoded in the spatial domain are non-all zero; or

[0015] A first reference block whose corresponding residual of the block to be encoded in the time domain is non-all zero, and two second reference blocks whose corresponding residual of the block to be encoded in the spatial domain is non-all zero; or

[0016] The corresponding residuals of the block to be encoded in the time domain are two first reference blocks that are not all zero; or,

[0017] The corresponding residuals of the block to be encoded in the spatial domain are two second reference blocks that are not all zero.

[0018] In at least one exemplary embodiment, based on the residual of the reference block of the block to be encoded, the residual of the block to be encoded is predicted to obtain the predicted residual of the block to be encoded, including: inputting the residual of the reference block into a residual prediction model to obtain the predicted residual of the block to be encoded, wherein the residual prediction model is obtained by training a deep learning network based on training samples, and the training samples include: the residual of the reference block of the coding block with known residual, and the actual residual of the coding block with known residual; or, linearly weighting the residual of the reference block to obtain the predicted residual of the block to be encoded, wherein the linear weighting includes linear weighting with a single weight or linear weighting with multiple weights.

[0019] In at least one exemplary embodiment, the actual residual of the block to be encoded is a difference between a pixel value of an original image block of the block to be encoded and a pixel value of a prediction block of the block to be encoded, wherein the prediction block of the block to be encoded is a block obtained after motion compensation is performed on the reference block of the encoding block.

[0020] According to another embodiment of the present invention, a residual decoding method is provided, comprising: determining a prediction block of a block to be decoded based on a motion vector MV parsed from a code stream, and obtaining a residual of a reference block of the block to be decoded, wherein the reference block of the block to be decoded comprises: at least two first reference blocks of the block to be decoded in the time domain; or, at least two second reference blocks of the block to be decoded in the spatial domain; or, at least one first reference block of the block to be decoded in the time domain and at least one second reference block of the block to be decoded in the spatial domain; predicting the residual of the block to be decoded based on the residual of the reference block of the block to be decoded to obtain a prediction residual of the block to be decoded; determining the actual residual of the block to be decoded according to a residual difference between the prediction residual of the block to be decoded and the prediction residual of the block to be decoded parsed from the code stream and the actual residual of the block to be decoded.

[0021] In at least one exemplary embodiment, when the image frame where the block to be decoded is located is a P frame, the first reference block of the block to be decoded in the time domain includes: the optimal prediction unit block PU of the block to be decoded in the forward reference frame, recorded as the first optimal PU, and / or, the optimal PU of the block to be decoded in the previous frame of the forward reference frame, recorded as the second optimal PU; or, when the image frame where the block to be decoded is located is a B frame, the first reference block of the block to be decoded in the time domain includes: the optimal PU of the block to be decoded in the forward reference frame, recorded as the third optimal PU, and / or, the optimal PU of the block to be decoded in the backward reference frame, recorded as the fourth optimal PU.

[0022] In at least one exemplary embodiment, when the image frame in which the block to be decoded is located is a P frame, the first reference block of the block to be decoded in the time domain is determined in the following manner: determining the first optimal PU of the block to be decoded in the forward reference frame according to the MV parsed from the code stream; determining the second optimal PU in the previous frame of the forward reference frame through motion estimation according to the co-located PU of the first optimal PU in the previous frame of the forward reference frame.

[0023] In at least one exemplary embodiment, the second reference block of the block to be decoded in the spatial domain is located in the image frame where the block to be decoded is located, and is adjacent to the block to be decoded in the spatial domain.

[0024] In at least one exemplary embodiment, the second reference block of the block to be decoded in the spatial domain includes: a left block and / or an upper block adjacent to the block to be decoded in the image frame where the block to be decoded is located.

[0025] In at least one exemplary embodiment, predicting the residual of the block to be decoded based on the residual of the reference block of the block to be decoded to obtain the predicted residual of the block to be decoded includes: inputting the residual of the reference block into a residual prediction model to obtain the predicted residual of the block to be decoded, wherein the residual prediction model is the same as the residual prediction model on the encoder side; or linearly weighting the residual of the reference block to obtain the predicted residual of the block to be decoded, wherein the linear weighting is the same as the linear weighting on the encoder side.

[0026] In at least one exemplary embodiment, after determining the actual residual of the block to be decoded by using the residual difference between the prediction residual of the block to be decoded and the prediction residual of the block to be decoded parsed from the code stream and the actual residual of the block to be decoded, the method further includes: adding the actual residual to the prediction block of the block to be decoded to restore the original image block of the block to be decoded.

[0027] According to another embodiment of the present invention, a residual coding device is provided, comprising: an encoder-side reference residual determination module, configured to determine the residual of a reference block of a block to be coded, wherein the reference block of the block to be coded comprises: at least two first reference blocks of the block to be coded in the time domain; or, at least two second reference blocks of the block to be coded in the spatial domain; or, at least one first reference block of the block to be coded in the time domain and at least one second reference block of the block to be coded in the spatial domain; an encoder-side residual prediction module, configured to predict the residual of the block to be coded based on the residual of the reference block of the block to be coded, to obtain the predicted residual of the block to be coded; and a residual coding module, configured to encode the residual difference between the predicted residual of the block to be coded and the actual residual of the block to be coded into a bitstream.

[0028] According to another embodiment of the present invention, a residual decoding device is provided, comprising: a decoder-side reference residual determination module, configured to determine a prediction block of a block to be decoded based on a motion vector MV parsed from a bitstream, and obtain a residual of the reference block of the block to be decoded, wherein the reference block of the block to be decoded includes: at least two first reference blocks of the block to be decoded in the time domain; or, at least two second reference blocks of the block to be decoded in the spatial domain; or, at least one first reference block of the block to be decoded in the time domain and at least one second reference block of the block to be decoded in the spatial domain; a decoder-side residual prediction module, configured to predict the residual of the block to be decoded based on the residual of the reference block of the block to be decoded, and obtain a prediction residual of the block to be decoded; and a residual decoding module, configured to determine the actual residual of the block to be decoded according to a residual difference between the prediction residual of the block to be decoded and the prediction residual of the block to be decoded parsed from the bitstream and the actual residual of the block to be decoded.

[0029] According to yet another embodiment of the present invention, a storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when running.

[0030] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0031] Through the present invention, the residual of the block to be encoded is predicted based on the residual of the reference block to obtain the predicted residual, and the residual difference between the predicted residual and the actual residual is encoded into the bitstream. Because the encoder only needs to transmit the difference between the actual residual of the current block to be encoded and the predicted residual, and the size of the residual difference is much lower than the actual residual, the bit rate required for the encoding residual is effectively reduced, achieving the effect of saving the bit rate. The same residual prediction process as the encoder is arranged on the decoder side. The decoder side predicts the residual of the block to be decoded based on the residual of the reference block of the block to be decoded, and determines the actual residual of the block to be decoded based on the predicted residual and the residual difference carried in the bitstream, which can ensure that on the decoder side, the actual residual of the block to be decoded can still be correctly restored based on the bitstream that saves the bit rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0033] Figure 1 is a hardware structure block diagram of a mobile terminal 10 provided with an inter-frame prediction module capable of applying a residual encoding method and a residual decoding method according to an embodiment of the present invention;

[0034] Figure 2 is a flowchart of a residual encoding method according to embodiment 1 of the present invention;

[0035] Figure 3 is a structural block diagram of a residual coding apparatus according to Embodiment 1 of the present invention;

[0036] Figure 4 is a flowchart of a residual decoding method according to Embodiment 2 of the present invention;

[0037] Figure 5 is a structural block diagram of a residual decoding device according to Embodiment 2 of the present invention;

[0038] Figure 6 is a schematic diagram of a convolutional neural network for generating a residual prediction module according to Embodiment 3 of the present invention;

[0039] Figure 7 is a schematic diagram of selecting a temporal reference residual block in a P frame and a B frame according to Embodiment 3 of the present invention;

[0040] Figure 8 is a schematic diagram of selecting a spatial reference residual block according to Embodiment 3 of the present invention;

[0041] Fig. 9 It is a schematic diagram of a method for improving video inter-frame coding performance by utilizing spatiotemporal correlation according to embodiment 3 of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0043] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0044] The embodiment of the present invention provides a scheme for improving coding performance (for example, video inter-frame coding performance) by using temporal and spatial correlation. The scheme uses a residual prediction module in video inter-frame prediction to predict the residual value of a coding block to reduce the bit rate of the coding residual, and is used to improve the inter-frame prediction coding efficiency in video coding. The codec adds a residual prediction module to the inter-frame prediction, and predicts the residual value of the current block to be coded (hereinafter referred to as the prediction residual) by using the residual of the reference block in the spatial domain and temporal domain of the current block to be coded. Compared with the coding scheme of directly encoding the residual block of the block to be coded into the bitstream, the encoder only needs to transmit the difference between the actual residual of the current block and the predicted residual, thereby reducing the bit rate required for the coding residual, so as to achieve the effect of saving the bit rate.

[0045] The following embodiment 1 describes a residual coding scheme from the encoder side that can reduce the bit rate required for encoding residuals, and embodiment 2 of the present application describes a residual decoding scheme for correctly parsing the residual block of the block to be decoded corresponding to the encoder side from the decoder side. The encoder side embodiment provided in embodiment 1 and the decoder side embodiment provided in embodiment 2 can be respectively applied to the inter-frame prediction module of the codec side (for example, it can be applied to the inter-frame prediction module of the existing video coding standard), and these inter-frame prediction modules can be set in a mobile terminal, a computer terminal or a similar computing device. Taking the inter-frame prediction module set on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal 10 provided with an inter-frame prediction module capable of applying a residual encoding method and a residual decoding method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal 10 may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA, on which a corresponding program may be run to implement the function of an inter-frame prediction module capable of applying a residual encoding method and a residual decoding method) and a memory 104 for storing data. Optionally, the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1The structure shown is for illustration only and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0046] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer programs corresponding to the embodiments of the encoder end provided in Embodiment 1 of the present invention and the embodiments of the decoder end provided in Embodiment 2. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, to implement the above-mentioned method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0047] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0048] The residual coding scheme is described below from the encoder side and the decoder side through embodiments 1 and 2 respectively.

[0049] Example 1

[0050] In this embodiment, a residual coding method is provided. Figure 2 is a flowchart of a residual coding method according to Embodiment 1 of the present invention, such as Figure 2 As shown, the process includes the following steps:

[0051] Step S202, determining a residual of a reference block of a block to be encoded (in an embodiment of the present invention, this term is also referred to as a residual, a residual block, a reference residual, or a reference residual block), wherein the reference block of the block to be encoded includes: at least two first reference blocks of the block to be encoded in the time domain; or, at least two second reference blocks of the block to be encoded in the spatial domain; or, at least one first reference block of the block to be encoded in the time domain and at least one second reference block of the block to be encoded in the spatial domain;

[0052] Step S204, predicting the residual of the block to be encoded based on the residual of the reference block of the block to be encoded, to obtain a predicted residual of the block to be encoded;

[0053] Step S206: Encode the residual difference between the prediction residual of the block to be encoded and the actual residual of the block to be encoded into a bitstream (eg, a video inter-frame encoding bitstream).

[0054] The residual of the reference block mentioned in step S202 refers to the actual coding residual of the reference block itself obtained according to the traditional method during the encoding process. Because the reference block has been encoded, the residual of the reference block can be obtained through technical means.

[0055] The actual residual of the block to be encoded mentioned in step S206 is the difference between the pixel value of the original image block of the block to be encoded and the pixel value of the prediction block of the block to be encoded, wherein the prediction block of the block to be encoded is a block obtained after motion compensation is performed on the reference block of the block to be encoded.

[0056] Through the above steps, the residual of the block to be coded is predicted according to the residual of the reference block to obtain the predicted residual, and the residual difference between the predicted residual and the actual residual is encoded into the bitstream. Because the encoder only needs to transmit the difference between the actual residual of the current block to be coded and the predicted residual, and the size of the residual difference is much lower than the actual residual, the bit rate required for coding the residual is effectively reduced, achieving the effect of saving bit rate.

[0057] Regarding the selection of the reference block of the block to be encoded, in view of the continuity of the video image, selecting a suitable reference block can accurately predict the residual of the block to be encoded.

[0058] In at least one exemplary embodiment, when the image frame in which the block to be encoded is located is a P frame, the first reference block of the block to be encoded in the time domain may include: an optimal prediction unit block PU of the block to be encoded in a forward reference frame, recorded as a first optimal PU, and / or an optimal PU of the block to be encoded in a frame previous to the forward reference frame, recorded as a second optimal PU.

[0059] When the image frame where the block to be encoded is located is a B frame, the first reference block of the block to be encoded in the time domain includes: the optimal PU of the block to be encoded in the forward reference frame, recorded as the third optimal PU, and / or the optimal PU of the block to be encoded in the backward reference frame, recorded as the fourth optimal PU.

[0060] In at least one exemplary embodiment, when the image frame in which the block to be encoded is located is a P frame, the first reference block of the block to be encoded in the time domain can be determined in the following manner: determining the first optimal PU of the block to be encoded in the forward reference frame through motion estimation, and determining the motion vector MV of the block to be encoded relative to the first optimal PU; determining the second optimal PU in a frame previous to the forward reference frame through motion estimation according to the position of the first optimal PU and the MV.

[0061] In at least one exemplary embodiment, the second reference block of the block to be encoded in the spatial domain is located in the image frame where the block to be encoded is located, and is adjacent to the block to be encoded in the spatial domain.

[0062] In at least one exemplary embodiment, the second reference block of the block to be encoded in the spatial domain includes: a block to the left and / or a block above the block to be encoded in the image frame where the block to be encoded is located (preferably, it can be a block to the left and / or a block above the block adjacent to the block to be encoded in the spatial domain with the same size).

[0063] Those skilled in the art should understand that, in practical applications, the first reference block in the time domain of the block to be encoded and / or the second reference block in the spatial domain of the block to be encoded can be determined in the above manner. The residual corresponding to the first reference block in the time domain can be called a time domain reference residual block, and the residual corresponding to the second reference block in the spatial domain can be called a spatial domain reference residual block. Based on at least one time domain reference residual block and / or at least one spatial domain reference residual block, the residual information of the block to be encoded can be predicted.

[0064] In a preferred implementation, in order to consider data symmetry, two time domain reference residual blocks and two spatial domain reference residual blocks can be selected, and residual prediction is performed based on these four reference residual blocks. At this time, the reference blocks of the block to be encoded include: two first reference blocks whose corresponding residuals of the block to be encoded in the time domain are non-all zero, and two second reference blocks whose corresponding residuals of the block to be encoded in the spatial domain are non-all zero.

[0065] In another preferred embodiment, when only one time domain reference residual block and one spatial domain reference residual block are non-all-zero blocks, one time domain and one spatial domain reference residual block can be selected, and residual prediction can be performed based on the two reference residual blocks. In this case, the reference block of the block to be encoded includes: a first reference block whose corresponding residual in the time domain is non-all-zero, and a second reference block whose corresponding residual in the spatial domain is non-all-zero.

[0066] In another preferred embodiment, when only two time domain reference residual blocks and one spatial domain reference residual block are non-all-zero blocks, two time domain and one spatial domain reference residual blocks can be selected, and residual prediction is performed based on these three reference residual blocks. In this case, the reference blocks of the block to be encoded include: two first reference blocks whose corresponding residuals in the time domain are non-all-zero, and a second reference block whose corresponding residual in the spatial domain is non-all-zero.

[0067] In another preferred embodiment, when only one time domain reference residual block and two spatial domain reference residual blocks are non-all zero blocks, one time domain and two spatial domain reference residual blocks can be selected, and residual prediction can be performed based on these three reference residual blocks. In this case, the reference block of the block to be encoded includes: a first reference block whose corresponding residual in the time domain is non-all zero, and two second reference blocks whose corresponding residual in the spatial domain is non-all zero.

[0068] In another preferred embodiment, when only two time domain reference residual blocks in the time domain are non-all-zero blocks, two time domain reference residual blocks can be selected, and residual prediction is performed based on the two time domain reference residual blocks. In this case, the reference block of the block to be encoded includes: two first reference blocks whose corresponding residuals in the time domain of the block to be encoded are non-all-zero.

[0069] In another preferred embodiment, when only two reference residual blocks in the spatial domain are non-all-zero blocks, two spatial reference residual blocks can be selected, and residual prediction is performed based on the two spatial reference residual blocks. In this case, the reference block of the block to be encoded includes: two second reference blocks whose corresponding residuals in the spatial domain of the block to be encoded are non-all-zero.

[0070] The process of predicting the residual of the block to be encoded based on the residual of the reference block can be implemented in a variety of prediction methods. In at least one exemplary embodiment, predicting the residual of the block to be encoded based on the residual of the reference block of the block to be encoded, and obtaining the predicted residual of the block to be encoded may include one of the following:

[0071] (1) inputting the residual of the reference block into a residual prediction model to obtain a prediction residual of the block to be encoded, wherein the residual prediction model may be a residual prediction model trained by a deep learning network, that is, it is obtained by training the deep learning network based on training samples, and the training samples include: the residual of the reference block of the encoding block with a known residual, and the actual residual of the encoding block with the known residual;

[0072] (2) Linearly weighting the residual of the reference block to obtain a predicted residual of the block to be encoded, wherein the linear weighting includes linear weighting with a single weight or linear weighting with multiple weights.

[0073] For example, the linear weighted sum of a single weight can be calculated using the following formula:

[0074] ReiPred(i, j) = W 1 ResiA(i,j)+W 2 ResiB(i,j)

[0075] W1 and W2 are weights, ResiA(i, j), ResiB(i, j) are the pixel values ​​of the selected reference residual block at the pixel point (i, j), and ReiPred(i, j) is the pixel value of the predicted residual block at the pixel point (i, j).

[0076] For example, the linear weighted sum of multiple weights can be calculated using the following formula:

[0077]

[0078] W1 ij and W2 ij is the weight value corresponding to each pixel of the reference residual block, which can be obtained through training. ResiA(i, j), ResiB(i, j) are the pixel values ​​of the selected reference residual block at pixel point (i, j), and ReiPred(i, j) is the pixel value of the predicted residual block at pixel point (i, j).

[0079] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0080] In this embodiment, a residual coding device is provided corresponding to the above-mentioned residual coding method, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0081] Figure 3 is a structural block diagram of a residual coding apparatus according to Embodiment 1 of the present invention, such as Figure 3 As shown, the device comprises:

[0082] The encoder-side reference residual determination module 32 is configured to determine the residual of a reference block of a block to be encoded, wherein the reference block of the block to be encoded includes: at least two first reference blocks of the block to be encoded in the time domain; or, at least two second reference blocks of the block to be encoded in the spatial domain; or, at least one first reference block of the block to be encoded in the time domain and at least one second reference block of the block to be encoded in the spatial domain;

[0083] The encoder-side residual prediction module 34 is configured to predict the residual of the block to be encoded based on the residual of the reference block of the block to be encoded, so as to obtain the predicted residual of the block to be encoded;

[0084] The residual encoding module 36 is configured to encode the residual difference between the predicted residual of the block to be encoded and the actual residual of the block to be encoded into a bitstream.

[0085] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0086] This embodiment further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of the above residual encoding method in this embodiment when running.

[0087] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0088] S1, determining a residual of a reference block of a block to be encoded (in an embodiment of the present invention, this term is also referred to as a residual, a residual block, a reference residual, or a reference residual block), wherein the reference block of the block to be encoded includes: at least two first reference blocks of the block to be encoded in the time domain; or, at least two second reference blocks of the block to be encoded in the spatial domain; or, at least one first reference block of the block to be encoded in the time domain and at least one second reference block of the block to be encoded in the spatial domain;

[0089] S2, predicting the residual of the block to be encoded based on the residual of the reference block of the block to be encoded, to obtain a predicted residual of the block to be encoded;

[0090] S3: Encode the residual difference between the prediction residual of the block to be encoded and the actual residual of the block to be encoded into a bitstream (for example, a video inter-frame encoding bitstream).

[0091] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0092] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps of the residual encoding method in the embodiment.

[0093] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0094] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0095] S1, determining a residual of a reference block of a block to be encoded (in an embodiment of the present invention, this term is also referred to as a residual, a residual block, a reference residual, or a reference residual block), wherein the reference block of the block to be encoded includes: at least two first reference blocks of the block to be encoded in the time domain; or, at least two second reference blocks of the block to be encoded in the spatial domain; or, at least one first reference block of the block to be encoded in the time domain and at least one second reference block of the block to be encoded in the spatial domain;

[0096] S2, predicting the residual of the block to be encoded based on the residual of the reference block of the block to be encoded, to obtain a predicted residual of the block to be encoded;

[0097] S3: Encode the residual difference between the prediction residual of the block to be encoded and the actual residual of the block to be encoded into a bitstream (for example, a video inter-frame encoding bitstream).

[0098] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0099] Example 2

[0100] In this embodiment, a residual decoding method is provided. Figure 4 is a flowchart of a residual decoding method according to Embodiment 2 of the present invention, such as Figure 4 As shown, the process includes the following steps:

[0101] Step S402, determining a prediction block of a block to be decoded based on a motion vector MV in a code stream (for example, a video inter-frame coding code stream), and obtaining a residual (in an embodiment of the present invention, this term is also referred to as a residual, a residual block, a reference residual, or a reference residual block) of a reference block of the block to be decoded, wherein the reference block of the block to be decoded includes: at least two first reference blocks of the block to be decoded in the time domain; or, at least two second reference blocks of the block to be decoded in the spatial domain; or, at least one first reference block of the block to be decoded in the time domain and at least one second reference block of the block to be decoded in the spatial domain;

[0102] Step S404, predicting the residual of the block to be decoded based on the residual of the reference block of the block to be decoded, to obtain a predicted residual of the block to be decoded;

[0103] Step S406: determining the actual residual of the block to be decoded according to the prediction residual of the block to be decoded and the residual difference between the prediction residual of the block to be decoded parsed from the bitstream and the actual residual of the block to be decoded.

[0104] The residual of the reference block mentioned in step S402 refers to the actual coding residual of the reference block itself obtained according to the traditional method during the encoding process. Because the reference block has been encoded, the residual of the reference block can be obtained through technical means.

[0105] In step S402, the prediction block of the block to be decoded is a block obtained after motion compensation is performed on the block to be decoded.

[0106] The actual residual of the block to be decoded mentioned in step S406 is the difference between the pixel value of the original image block of the block to be decoded and the pixel value of the prediction block of the block to be decoded.

[0107] Through the above steps, since the same residual prediction process as that of the encoder is arranged at the decoder end, the decoder end predicts the residual of the block to be decoded based on the residual of the reference block of the block to be decoded, and determines the actual residual of the block to be decoded according to the predicted residual and the residual difference value carried in the bitstream, it can be ensured that at the decoder end, the actual residual of the block to be decoded can still be correctly restored based on the bitstream with saved bitrate.

[0108] In at least one exemplary embodiment, when the image frame in which the block to be decoded is located is a P frame, the first reference block of the block to be decoded in the time domain includes: an optimal prediction unit block PU of the block to be decoded in a forward reference frame, recorded as a first optimal PU, and / or an optimal PU of the block to be decoded in a frame previous to the forward reference frame, recorded as a second optimal PU.

[0109] In at least one exemplary embodiment, when the image frame in which the block to be decoded is located is a B frame, the first reference block of the block to be decoded in the time domain includes: the optimal PU of the block to be decoded in the forward reference frame, recorded as the third optimal PU, and / or the optimal PU of the block to be decoded in the backward reference frame, recorded as the fourth optimal PU.

[0110] It should be noted that, in order to ensure the accuracy of residual prediction, the same reference block selection method as that of the encoder is adopted at the decoder to ensure consistency with the prediction residual obtained by residual prediction at the encoder.

[0111] In at least one exemplary embodiment, when the image frame in which the block to be decoded is located is a P frame, the first reference block of the block to be decoded in the time domain is determined in the following manner: determining the first optimal PU of the block to be decoded in the forward reference frame according to the MV parsed from the code stream; determining the second optimal PU in the previous frame of the forward reference frame through motion estimation according to the co-located PU of the first optimal PU in the previous frame of the forward reference frame.

[0112] In at least one exemplary embodiment, the second reference block of the block to be decoded in the spatial domain is located in the image frame where the block to be decoded is located, and is adjacent to the block to be decoded in the spatial domain.

[0113] In at least one exemplary embodiment, the second reference block of the block to be decoded in the spatial domain includes: a block to the left and / or a block above the block to be decoded in the image frame where the block to be decoded is located (preferably, it can be a block to the left and / or a block above the block to be decoded that is adjacent to the block to be decoded in the spatial domain with the same size).

[0114] Those skilled in the art should understand that, in practical applications, the first reference block in the time domain of the block to be decoded and / or the second reference block in the spatial domain of the block to be decoded can be determined in the above manner. The residual corresponding to the first reference block in the time domain can be called a time domain reference residual block, and the residual corresponding to the second reference block in the spatial domain can be called a spatial domain reference residual block. Based on at least one time domain reference residual block and / or at least one spatial domain reference residual block, the residual information of the block to be decoded can be predicted.

[0115] In a preferred implementation, in order to consider data symmetry, two time domain reference residual blocks and two spatial domain reference residual blocks can be selected, and residual prediction is performed based on these four reference residual blocks. At this time, the reference blocks of the block to be decoded include: two first reference blocks whose corresponding residuals of the block to be decoded in the time domain are not all zero, and two second reference blocks whose corresponding residuals of the block to be decoded in the spatial domain are not all zero.

[0116] In another preferred embodiment, when only one time domain reference residual block and one spatial domain reference residual block are non-all zero blocks, one time domain and one spatial domain reference residual block can be selected, and residual prediction can be performed based on the two reference residual blocks. In this case, the reference block of the block to be decoded includes: a first reference block whose corresponding residual in the time domain of the block to be decoded is non-all zero, and a second reference block whose corresponding residual in the spatial domain of the block to be decoded is non-all zero.

[0117] In another preferred embodiment, when only two time domain reference residual blocks and one spatial domain reference residual block are non-all-zero blocks, two time domain and one spatial domain reference residual blocks can be selected, and residual prediction can be performed based on these three reference residual blocks. In this case, the reference block of the block to be decoded includes: two first reference blocks whose corresponding residuals in the time domain of the block to be decoded are non-all-zero, and a second reference block whose corresponding residual in the spatial domain of the block to be decoded is non-all-zero.

[0118] In another preferred implementation, when only one time domain reference residual block and two spatial domain reference residual blocks are non-all-zero blocks, one time domain and two spatial domain reference residual blocks can be selected, and residual prediction can be performed based on these three reference residual blocks. In this case, the reference block of the block to be decoded includes: a first reference block whose corresponding residual in the time domain of the block to be decoded is non-all-zero, and two second reference blocks whose corresponding residual in the spatial domain of the block to be decoded is non-all-zero.

[0119] In another preferred implementation, when only two time domain reference residual blocks in the time domain are non-all-zero blocks, two time domain reference residual blocks can be selected, and residual prediction is performed based on the two time domain reference residual blocks. In this case, the reference block of the block to be decoded includes: two first reference blocks whose corresponding residuals in the time domain of the block to be decoded are non-all-zero.

[0120] In another preferred implementation, when only two reference residual blocks in the spatial domain are non-all-zero blocks, two spatial reference residual blocks can be selected, and residual prediction is performed based on the two spatial reference residual blocks. In this case, the reference block of the block to be decoded includes: two second reference blocks whose corresponding residuals in the spatial domain of the block to be decoded are non-all-zero.

[0121] The process of predicting the residual of the block to be decoded based on the residual of the reference block can be implemented in a variety of prediction methods. In at least one exemplary embodiment, predicting the residual of the block to be decoded based on the residual of the reference block of the block to be decoded, and obtaining the predicted residual of the block to be decoded can include one of the following:

[0122] (1) inputting the residual of the reference block into a residual prediction model to obtain a prediction residual of the block to be decoded, wherein the residual prediction model is the same as the residual prediction model at the encoder end;

[0123] (2) Linearly weighting the residual of the reference block to obtain a prediction residual of the block to be decoded, wherein the linear weighting is the same as the linear weighting at the encoder end.

[0124] For example, the linear weighted sum of a single weight can be calculated using the following formula:

[0125] ReiPred(i, j) = W 1 ResiA(i,j)+W 2 ResiB(i,j)

[0126] W1 and W2 are weights, ResiA(i, j), ResiB(i, j) are the pixel values ​​of the selected reference residual block at the pixel point (i, j), and ReiPred(i, j) is the pixel value of the predicted residual block at the pixel point (i, j).

[0127] For example, the linear weighted sum of multiple weights can be calculated using the following formula:

[0128]

[0129] W1 ij and W2 ijis the weight value corresponding to each pixel of the reference residual block, which can be obtained through training. ResiA(i, j), ResiB(i, j) are the pixel values ​​of the selected reference residual block at pixel point (i, j), and ReiPred(i, j) is the pixel value of the predicted residual block at pixel point (i, j).

[0130] In at least one exemplary embodiment, after determining the actual residual of the block to be decoded by using the residual difference between the prediction residual of the block to be decoded and the prediction residual of the block to be decoded parsed from the code stream and the actual residual of the block to be decoded, the method further includes: adding the actual residual to the prediction block of the block to be decoded to restore the original image block of the block to be decoded.

[0131] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0132] In this embodiment, a residual decoding device is provided corresponding to the above residual decoding method, which is used to implement the above embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0133] Figure 5 is a structural block diagram of a residual decoding device according to Embodiment 2 of the present invention, such as Figure 5 As shown, the device comprises:

[0134] The decoder-side reference residual determination module 52 is configured to determine a prediction block of a block to be decoded based on a motion vector MV parsed from a bitstream, and obtain a residual of a reference block of the block to be decoded, wherein the reference block of the block to be decoded includes: at least two first reference blocks of the block to be decoded in the time domain; or, at least two second reference blocks of the block to be decoded in the spatial domain; or, at least one first reference block of the block to be decoded in the time domain and at least one second reference block of the block to be decoded in the spatial domain;

[0135] A decoder-side residual prediction module 54 is configured to predict the residual of the block to be decoded based on the residual of the reference block of the block to be decoded, so as to obtain a predicted residual of the block to be decoded;

[0136] The residual decoding module 56 is configured to determine the actual residual of the block to be decoded according to the prediction residual of the block to be decoded and the residual difference between the prediction residual of the block to be decoded parsed from the bitstream and the actual residual of the block to be decoded.

[0137] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0138] This embodiment further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of the residual decoding method in this embodiment when running.

[0139] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0140] S1, determining a prediction block of a block to be decoded based on a motion vector MV in a code stream (for example, a video inter-frame coding code stream), and obtaining a residual (in an embodiment of the present invention, this term is also referred to as a residual, a residual block, a reference residual, or a reference residual block) of a reference block of the block to be decoded, wherein the reference block of the block to be decoded includes: at least two first reference blocks of the block to be decoded in the time domain; or, at least two second reference blocks of the block to be decoded in the spatial domain; or, at least one first reference block of the block to be decoded in the time domain and at least one second reference block of the block to be decoded in the spatial domain;

[0141] S2, predicting the residual of the block to be decoded based on the residual of the reference block of the block to be decoded, to obtain a predicted residual of the block to be decoded;

[0142] S3, determining the actual residual of the block to be decoded according to the prediction residual of the block to be decoded and the residual difference between the prediction residual of the block to be decoded parsed from the bitstream and the actual residual of the block to be decoded.

[0143] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0144] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps of the above-mentioned residual decoding method in this embodiment.

[0145] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0146] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0147] S1, determining a prediction block of a block to be decoded based on a motion vector MV in a code stream (for example, a video inter-frame coding code stream), and obtaining a residual (in an embodiment of the present invention, this term is also referred to as a residual, a residual block, a reference residual, or a reference residual block) of a reference block of the block to be decoded, wherein the reference block of the block to be decoded includes: at least two first reference blocks of the block to be decoded in the time domain; or, at least two second reference blocks of the block to be decoded in the spatial domain; or, at least one first reference block of the block to be decoded in the time domain and at least one second reference block of the block to be decoded in the spatial domain;

[0148] S2, predicting the residual of the block to be decoded based on the residual of the reference block of the block to be decoded, to obtain a predicted residual of the block to be decoded;

[0149] S3, determining the actual residual of the block to be decoded according to the prediction residual of the block to be decoded and the residual difference between the prediction residual of the block to be decoded parsed from the bitstream and the actual residual of the block to be decoded.

[0150] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0151] The following embodiment 3 takes the residual prediction model (also referred to as the residual prediction module) trained by a deep learning network as an example to describe in detail the specific implementation of the residual coding and decoding scheme. It should be noted that for the scheme of using a linear weighted sum method for residual prediction, the principle of the overall residual coding and decoding is similar to the following embodiments and will not be repeated.

[0152] Example 3

[0153] This embodiment designs a method for improving the inter-frame coding performance of video by using temporal and spatial correlation for the coding unit of video coding. In actual use, the encoder will call this method to complete inter-frame prediction. The following is a detailed description of the contents involved in this method.

[0154] (1) Residual prediction module

[0155] The residual prediction module can be generated by a convolutional neural network. Figure 6 is a schematic diagram of a convolutional neural network for generating a residual prediction module according to Embodiment 3 of the present invention, such as Figure 6 As shown, the convolutional neural network takes two time domain reference residual blocks and two spatial domain reference residual blocks related to the current coding block as input (it can also be other numbers of time domain and spatial domain reference residual blocks, or only time domain reference residual blocks or only spatial domain reference residual blocks), extracts the feature information of the residual image through convolution and pooling, and then outputs the predicted residual through the deconvolution process.

[0156] The input of the convolutional neural network is four reference residual blocks, two selected from the temporal reference frame and two selected from the adjacent spatial domain.

[0157] In the time domain, different operations are performed according to the different prediction methods of P frames and B frames. Figure 7 is a schematic diagram of selecting a temporal reference residual block in a P frame and a B frame according to Embodiment 3 of the present invention, such as Figure 7 As shown:

[0158] For P frames, since it is a unidirectional prediction and has only one forward reference frame, firstly, the optimal prediction unit block (Prediction Unit, PU) is found in the forward reference frame through motion estimation, and the motion vector (motion vector, MV) of the current coding block is generated. Then, based on the position of the optimal reference PU of the current coding block and the MV of the current coding block, another optimal reference PU is searched from the previous frame of the forward reference frame through motion estimation, and the residual blocks corresponding to the two optimal reference PUs are input as the two time domain reference residual blocks of the P frame.

[0159] For a B frame, there is a forward reference frame and a backward reference frame. A residual block of an optimal PU is found in each of the forward reference frame and the backward reference frame through motion estimation as a time domain reference residual block input.

[0160] In the airspace, Figure 8 FIG. 4 is a schematic diagram of selecting a spatial reference residual block according to Embodiment 3 of the present invention. Figure 8 As shown, the left and top blocks adjacent to the current block and of the same size are selected as reference blocks, and the residual block corresponding to the reference block is used as the input of the spatial domain reference residual block.

[0161] The output of the residual prediction module is the predicted residual block of the current block. At the encoder end, the predicted residual block can be subtracted from the actual residual block to obtain the coded residual. At the decoder end, the predicted residual block plus the coded residual can restore the actual residual block.

[0162] (2) Inter-frame coding block residual prediction method based on spatiotemporal correlation

[0163] Fig. 9 This is a schematic diagram of a method for improving the inter-frame coding performance of video by utilizing spatiotemporal correlation according to embodiment 3 of the present invention. In this figure, taking the implementation of the residual prediction module through a neural network as an example, the residual coding and decoding operations at the encoder and decoder ends are shown, which are described in detail below.

[0164] Operation on the encoder side:

[0165] In the first step, the video sequence provided by the pre-coding standard is extracted to extract the residual data of four reference blocks and the residual data of the current block as training data. Figure 6 The constructed deep neural network embodiment performs network training, and then embeds the trained deep neural network into the inter-frame prediction of the encoder.

[0166] The second step is to read the block to be encoded from the input video image, obtain the predicted block of the block to be encoded through motion estimation prediction, and subtract the pixel value of the predicted block from the pixel value of the original image to obtain the traditional residual block.

[0167] The third step is to search for the motion vector MV obtained by motion estimation according to the different prediction methods of P frame and B frame. Figure 7 The two time-domain reference residual blocks shown are then Figure 8 Two spatial reference residual blocks are obtained, and the obtained four reference residual blocks are input into the residual prediction module to obtain a predicted residual block.

[0168] The fourth step is to write the difference between the traditional residual block and the predicted residual block into the bitstream file as the coded residual of the current block to be coded. In this step, the traditional residual block is the residual value to be coded obtained by the traditional inter-frame prediction method for the current coding block, and the predicted residual is the residual value predicted by the residual prediction module.

[0169] In summary, we can see that at the encoder end, motion estimation is used to find the reference block in the reference frame, and after motion compensation operation, the prediction block is obtained. The difference between the original block and the prediction block is the actual residual block in the traditional inter-frame prediction mode. In this process, a residual prediction module is established to predict the residual of the current block. The input of the residual prediction module is two reference residual blocks in the time domain and the spatial domain corresponding to the reference block of the current coding block. The output of the residual prediction module is the predicted residual of the current block. The difference between the actual residual block and the predicted residual block is used as the coding residual for subsequent transformation, quantization and entropy coding.

[0170] Operation on the decoder side:

[0171] In the first step, the residual prediction module is embedded into the decoder inter-frame prediction.

[0172] The second step is to obtain the prediction block of the current block to be decoded through the motion vector MV parsed from the bitstream, and then find Figure 7 The two time domain reference residual blocks shown in Figure 8 The two spatial reference residual blocks shown are input into the residual prediction module to obtain the predicted residual blocks. In this step, the generation method of the four reference residual blocks corresponding to the current block is consistent with the four reference residual blocks at the encoder end.

[0173] The third step is to read the coded residual of the block to be decoded from the bitstream, then add the coded residual to the prediction residual block to restore the actual residual of the block to be decoded, and finally add the prediction block to restore the original image block.

[0174] In summary, we can see that in the inter-frame prediction decoding operation on the decoder side, the same residual prediction module as the encoder is established to predict the residual of the current block. The input of the residual prediction module is two reference residual blocks in the time domain and the spatial domain corresponding to the reference block of the current decoding block. The reference block is found in the same way as the encoder side, and the output is the prediction residual of the current decoding block. The decoded residual and the prediction block of the block to be decoded are added to complete the reconstruction of the current block.

[0175] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0176] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A residual coding method, characterized in that: include: Determine a residual of a reference block of a block to be encoded, wherein the reference block of the block to be encoded includes: at least two first reference blocks of the block to be encoded in the time domain; or, at least two second reference blocks of the block to be encoded in the spatial domain; or, at least one first reference block of the block to be encoded in the time domain and at least one second reference block of the block to be encoded in the spatial domain, wherein, in the case where the image frame in which the block to be encoded is located is a P frame, the first reference block of the block to be encoded in the time domain includes: an optimal prediction unit block PU of the block to be encoded in a forward reference frame, recorded as a first optimal PU, and an optimal PU of the block to be encoded in a frame previous to the forward reference frame, recorded as a second optimal PU; Based on the residual of the reference block of the block to be encoded, predicting the residual of the block to be encoded to obtain the predicted residual of the block to be encoded, wherein based on the residual of the reference block of the block to be encoded, predicting the residual of the block to be encoded to obtain the predicted residual of the block to be encoded comprises: inputting the residual of the reference block into a residual prediction model to obtain the predicted residual of the block to be encoded, wherein the residual prediction model is obtained by training with a deep learning network based on training samples, and the training samples include: the residual of the reference block of the coding block with known residual, and the actual residual of the coding block with known residual; The residual difference between the prediction residual of the block to be encoded and the actual residual of the block to be encoded is encoded into a bitstream.

2. The method according to claim 1, characterized in that When the image frame where the block to be encoded is located is a B frame, the first reference block of the block to be encoded in the time domain includes: the optimal PU of the block to be encoded in the forward reference frame, recorded as the third optimal PU, and / or the optimal PU of the block to be encoded in the backward reference frame, recorded as the fourth optimal PU.

3. The method according to claim 2, characterized in that In the case where the image frame where the block to be encoded is located is a P frame, the first reference block of the block to be encoded in the time domain is determined in the following manner: Determine the first optimal PU of the block to be encoded in the forward reference frame by motion estimation, and determine a motion vector MV of the block to be encoded relative to the first optimal PU; The second best PU is determined in a frame previous to the forward reference frame through motion estimation according to the position of the first best PU and the MV.

4. The method according to claim 1, characterized in that: The second reference block of the block to be encoded in the spatial domain is located in the image frame where the block to be encoded is located, and is adjacent to the block to be encoded in the spatial domain.

5. The method according to claim 4, characterized in that The second reference block of the block to be encoded in the spatial domain includes: a left block and / or an upper block adjacent to the block to be encoded in the image frame where the block to be encoded is located.

6. The method according to any one of claims 1 to 5, characterized in that The reference block of the block to be encoded comprises: Two first reference blocks whose corresponding residuals of the block to be encoded in the time domain are not all zero, and two second reference blocks whose corresponding residuals of the block to be encoded in the spatial domain are not all zero; or A first reference block whose corresponding residual of the block to be encoded in the time domain is non-all zero, and a second reference block whose corresponding residual of the block to be encoded in the spatial domain is non-all zero; or Two first reference blocks whose corresponding residuals of the block to be encoded in the time domain are non-all zero, and one second reference block whose corresponding residuals of the block to be encoded in the spatial domain are non-all zero; or A first reference block whose corresponding residual of the block to be encoded in the time domain is non-all zero, and two second reference blocks whose corresponding residual of the block to be encoded in the spatial domain is non-all zero; or The corresponding residuals of the block to be encoded in the time domain are two first reference blocks that are not all zero; or, The corresponding residuals of the block to be encoded in the spatial domain are two second reference blocks that are not all zero.

7. The method according to any one of claims 1 to 5, characterized in that The actual residual of the block to be encoded is the difference between the pixel values ​​of the original image block of the block to be encoded and the pixel values ​​of the prediction block of the block to be encoded, wherein the prediction block of the block to be encoded is a block obtained after motion compensation is performed on the reference block of the encoding block.

8. A residual decoding method, characterized in that: include: Determine a prediction block of a block to be decoded based on a motion vector MV parsed from a bitstream, and obtain a residual of a reference block of the block to be decoded, wherein the reference block of the block to be decoded includes: at least two first reference blocks of the block to be decoded in the time domain; or, at least two second reference blocks of the block to be decoded in the spatial domain; or, at least one first reference block of the block to be decoded in the time domain and at least one second reference block of the block to be decoded in the spatial domain, wherein, in the case where the image frame where the block to be decoded is located is a P frame, the first reference block of the block to be decoded in the time domain includes: an optimal prediction unit block PU of the block to be decoded in a forward reference frame, recorded as a first optimal PU, and an optimal PU of the block to be decoded in a frame previous to the forward reference frame, recorded as a second optimal PU; Based on the residual of the reference block of the block to be decoded, predicting the residual of the block to be decoded to obtain the predicted residual of the block to be decoded, wherein based on the residual of the reference block of the block to be decoded, predicting the residual of the block to be decoded to obtain the predicted residual of the block to be decoded comprises: inputting the residual of the reference block into a residual prediction model to obtain the predicted residual of the block to be decoded, wherein the residual prediction model is the same as the residual prediction model at the encoder end; The actual residual of the block to be decoded is determined according to the prediction residual of the block to be decoded and the residual difference value parsed from the bitstream, wherein the residual difference value is the difference between the prediction residual of the block to be decoded and the actual residual of the block to be decoded.

9. The method according to claim 8, characterized in that When the image frame where the block to be decoded is located is a B frame, the first reference block of the block to be decoded in the time domain includes: the optimal PU of the block to be decoded in the forward reference frame, recorded as the third optimal PU, and / or the optimal PU of the block to be decoded in the backward reference frame, recorded as the fourth optimal PU.

10. The method according to claim 9, characterized in that In the case where the image frame where the to-be-decoded block is located is a P frame, the first reference block of the to-be-decoded block in the time domain is determined in the following manner: Determine, according to the MV parsed from the bitstream, the first optimal PU of the block to be decoded in the forward reference frame; The second best PU is determined in the previous frame of the forward reference frame by motion estimation according to a co-located PU of the first best PU in the previous frame of the forward reference frame.

11. The method according to claim 8, characterized in that The second reference block of the block to be decoded in the spatial domain is located in the image frame where the block to be decoded is located, and is adjacent to the block to be decoded in the spatial domain.

12. The method according to claim 11, characterized in that The second reference block of the block to be decoded in the spatial domain includes: a left block and / or an upper block adjacent to the block to be decoded in the image frame where the block to be decoded is located.

13. The method according to any one of claims 8 to 12, characterized in that: After determining the actual residual of the block to be decoded by using the prediction residual of the block to be decoded and the residual difference value parsed from the bitstream, the method further includes: The actual residual is added to the predicted block of the block to be decoded to restore the original image block of the block to be decoded.

14. A residual coding device, characterized in that: include: The encoder-side reference residual determination module is configured to determine the residual of the reference block of the block to be encoded, wherein the reference block of the block to be encoded includes: at least two first reference blocks of the block to be encoded in the time domain; or, at least two second reference blocks of the block to be encoded in the spatial domain; or, at least one first reference block of the block to be encoded in the time domain and at least one second reference block of the block to be encoded in the spatial domain, wherein, in the case where the image frame where the block to be encoded is located is a P frame, the first reference block of the block to be encoded in the time domain includes: an optimal prediction unit block PU of the block to be encoded in a forward reference frame, recorded as a first optimal PU, and an optimal PU of the block to be encoded in a frame previous to the forward reference frame, recorded as a second optimal PU; An encoder-side residual prediction module is configured to predict the residual of the block to be encoded based on the residual of the reference block of the block to be encoded, so as to obtain the predicted residual of the block to be encoded, wherein the encoder-side residual prediction module is configured to input the residual of the reference block into a residual prediction model to obtain the predicted residual of the block to be encoded, wherein the residual prediction model is obtained by training with a deep learning network based on training samples, and the training samples include: the residual of the reference block of the coding block with a known residual, and the actual residual of the coding block with the known residual; The residual coding module is configured to encode the residual difference between the predicted residual of the block to be coded and the actual residual of the block to be coded into a bitstream.

15. A residual decoding device, characterized in that: include: A reference residual determination module at the decoder end is configured to determine a prediction block of a block to be decoded based on a motion vector MV parsed from a bitstream, and obtain a residual of the reference block of the block to be decoded, wherein the reference block of the block to be decoded includes: at least two first reference blocks of the block to be decoded in the time domain; or, at least two second reference blocks of the block to be decoded in the spatial domain; or, at least one first reference block of the block to be decoded in the time domain and at least one second reference block of the block to be decoded in the spatial domain, wherein, in the case where the image frame where the block to be decoded is located is a P frame, the first reference block of the block to be decoded in the time domain includes: an optimal prediction unit block PU of the block to be decoded in a forward reference frame, recorded as a first optimal PU, and an optimal PU of the block to be decoded in a frame previous to the forward reference frame, recorded as a second optimal PU; A residual prediction module at the decoder side is configured to predict the residual of the block to be decoded based on the residual of the reference block of the block to be decoded, so as to obtain the predicted residual of the block to be decoded, wherein the residual prediction module at the decoder side is configured to input the residual of the reference block into a residual prediction model to obtain the predicted residual of the block to be decoded, wherein the residual prediction model is the same as the residual prediction model at the encoder side; The residual decoding module is configured to determine the actual residual of the block to be decoded according to the prediction residual of the block to be decoded and the residual difference value parsed from the bitstream, wherein the residual difference value is the difference between the prediction residual of the block to be decoded and the actual residual of the block to be decoded.

16. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 13 when executed.

17. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 13.

Images