Encoding method and apparatus, and decoding method and apparatus

Abstract

Disclosed are an encoding method and apparatus, and a decoding method and apparatus, which relate to the technical field of multimedia. The decoding method comprises: receiving an encoded image bitstream, wherein the encoded image bitstream comprises a plurality of bitstream segments of a reference image and bitstreams of a plurality of display images, which bitstream segments and bitstreams are sorted in an interleaved manner; performing a decoding operation on the encoded image bitstream to acquire a reconstructed reference image; on the basis of the currently acquired reconstructed reference image, updating a decoded image buffer; and using the currently acquired reconstructed reference image as a reference image to perform inter-frame predictive decoding on the bitstreams of the display images. Thus, during inter-frame predictive decoding, upon acquiring a reconstructed reference image, a decoder updates a decoded image buffer on the basis of the currently acquired reconstructed reference image. It is not necessary to wait until a first image that only references a CRR image and is subsequent to the CRR image is received before reconstructed CRR images of CRR image bitstreams can be moved into the decoded image buffer. The decoded image buffer can be updated according to bitstream order, thereby reducing the system-level complexity.

Description

Encoding and decoding methods and devices Technical Field

[0001] This application relates to the field of media technology, and in particular to an encoding / decoding method and apparatus. Background Technology

[0002] With the widespread adoption of digital video applications (such as digital television broadcasting, video transmission over the internet, real-time video conferencing applications, video content storage, video content acquisition and editing systems, etc.), video encoding and decoding (video coding and video decoding) using codecs (encoders and decoders) are widely used. Video coding refers to the encoder compressing the original video stream using compression algorithms, resulting in a compressed video stream that is easier to store and transmit. Video decoding refers to the decoder decoding the compressed video stream to obtain a reconstructed video stream.

[0003] In related technologies, an encoding and decoding scheme based on Cross Random Access Point Reference (CRR) images is provided. For example, an encoder encodes slices of multiple display images and their corresponding CRR images to obtain an encoded image bitstream and a reconstructed CRR image. The encoded image bitstream includes multiple interleaved CRR image bitstream segments and multiple display image bitstreams. The encoder can use the reconstructed CRR image corresponding to the current display image as a reference image to perform inter-frame predictive coding on the current display image to eliminate temporal redundancy. Correspondingly, the decoder decodes the encoded image bitstream to obtain the reconstructed CRR image and multiple display image bitstreams. The decoder can use the reconstructed CRR image corresponding to the current display image bitstream as a reference image to perform inter-frame predictive decoding on the current display image bitstream to eliminate temporal redundancy. Since the CRR image corresponds to multiple display images, and these multiple display images can include multiple cross-random access points (RAPs), the reconstructed CRR image can be used as a reference across RAPs.

[0004] However, in the CRR slice refresh technique, the decoder can only move the reconstructed CRR image from the CRR image bitstream into the decoded image buffer and replace the previously reconstructed CRR image in the decoded image buffer if it is the first CRR image that references the CRR image after receiving the CRR image. This results in high system-level complexity and makes it impossible to update the decoded image buffer in bitstream order.

[0005] Summary of the Invention

[0006] This application provides an encoding / decoding method and apparatus that solves the problem in related technologies where the decoded image buffer cannot be updated according to the bit stream order, resulting in high system layer complexity.

[0007] In a first aspect, this application provides an encoding method, comprising: performing encoding operations on a reference image and a plurality of display images corresponding to the reference image to obtain an encoded image bitstream; acquiring a reconstructed reference image based on the encoding operations; updating a decoded image buffer according to the currently acquired reconstructed reference image after the bitstream transmission of the reference image included in the encoded image bitstream is completed; and performing inter-frame predictive coding on the bitstream of the display images using the currently acquired reconstructed reference image as the reference image. The encoded image bitstream includes: a bitstream of the reference image and bitstreams of the plurality of display images. The bitstream of the reference image precedes the bitstream of the reference start image corresponding to the reference image in the plurality of display image bitstreams. The reference start image includes the image of the first reference reference image among the plurality of display images.

[0008] When the encoder performs inter-frame predictive coding, it updates the decoded image buffer based on the reconstructed reference image as soon as it acquires the reconstructed reference image. It does not need to wait for the first image that only references the CRR image after receiving the CRR image before it can move the reconstructed CRR image of the CRR image bitstream into the decoded image buffer. The decoded image buffer can be updated according to the bitstream order, which reduces the complexity of the system layer.

[0009] In one possible implementation, the encoded image bitstream comprises n bitstream segments of a reference image. These n bitstream segments of the reference image are interleaved and ordered with the bitstreams of multiple display images. One or more bitstream segments of the reference image are inserted between two display images.

[0010] In one possible implementation, updating the decoded image buffer based on the currently acquired reconstructed reference image includes replacing the reconstructed reference image stored in the decoded image buffer with the currently acquired reconstructed reference image.

[0011] In this embodiment, if the encoder already contains a previously reconstructed CRR image, the previous reconstructed CRR image is replaced with a new one. This balances the ability to directly use the reconstructed image from the encoder for subsequent display images while improving update timeliness, thus achieving both accuracy and encoding efficiency.

[0012] In another possible implementation, the display image between the last bitstream segment of the n bitstream segments of the reference image and the reference enabled first image corresponding to the currently acquired reconstructed reference image cannot reference the knowledge image.

[0013] Alternatively, if there is an access unit for displaying an image between the access unit containing the last coded NAL unit of the knowledge image and the access unit containing the first RL image referencing the knowledge image, the display image cannot be decoded with reference to the knowledge image. An access unit is a group of NAL units that are interconnected according to specified rules and are sequential in decoding order.

[0014] In this embodiment of the application, the reference enabled first image corresponding to the currently acquired reconstructed reference image after the last bitstream segment is constrained to use the currently acquired reconstructed CRR image. Other display images after the last bitstream segment cannot refer to the knowledge image. This avoids encoding failure when the reconstructed CRR image in the decoding image buffer has been removed, making it impossible to refer to the reconstructed CRR image, and there is no currently effective reconstructed CRR image.

[0015] Secondly, this application provides a decoding method, which includes: receiving an encoded image bitstream, the encoded image bitstream including a reference image bitstream and multiple display image bitstreams. The reference image bitstream precedes the bitstream of a reference enabled first image corresponding to the reference image among the multiple display image bitstreams. The reference enabled first image includes the image of the first reference image among the multiple display images; performing a decoding operation on the encoded image bitstream, and based on the operation result, obtaining a reconstructed reference image, updating the decoded image buffer according to the currently obtained reconstructed reference image; and using the currently obtained reconstructed reference image as the reference image, performing inter-frame prediction decoding on the display image bitstream.

[0016] When the decoder performs inter-frame prediction decoding, it updates the decoded image buffer based on the reconstructed reference image as soon as it acquires the reconstructed reference image. It does not need to wait for the first image that only references the CRR image after receiving the CRR image before it can move the reconstructed CRR image of the CRR image bitstream into the decoded image buffer. The decoded image buffer can be updated according to the bitstream order, which reduces the complexity of the system layer.

[0017] In one possible implementation, the encoded image bitstream comprises n bitstream segments of a reference image. These n bitstream segments of the reference image are interleaved and ordered with the bitstreams of multiple display images. One or more bitstream segments of the reference image are inserted between two display images.

[0018] In one possible implementation, the encoded image bitstream includes n bitstream segments of a reference image; the encoded image bitstream is decoded, and a reconstructed reference image is obtained based on the result of the operation; the decoded image buffer is updated according to the currently obtained reconstructed reference image, including: after receiving n bitstream segments of the reference image, decoding the n bitstream segments of the reference image to obtain the currently obtained reconstructed reference image, and updating the decoded image buffer according to the currently obtained reconstructed reference image.

[0019] After receiving n bitstream segments of the reference image, the n bitstream segments of the reference image are decoded without the need for additional buffering to reconstruct the sub-image, thus saving storage space.

[0020] In another possible implementation, the encoded image bitstream includes n bitstream segments of a reference image; the encoded image bitstream is decoded, and a reconstructed reference image is obtained based on the result of the operation; the decoded image buffer is updated according to the currently obtained reconstructed reference image, including: receiving bitstream segments of the reference image, decoding the bitstream segments of the reference image to obtain reconstructed sub-images of the bitstream segments of the reference image; when the decoding of the n bitstream segments of the reference image is completed, the reconstructed sub-images of the n bitstream segments of the reference image are stitched together to obtain the currently obtained reconstructed reference image, and the decoded image buffer is updated according to the currently obtained reconstructed reference image.

[0021] The system receives bitstream segments of a reference image, decodes these segments, and after decoding the last bitstream segment, stitches together n bitstream segments of the reference image to create a reconstructed sub-image, thus obtaining the currently acquired reconstructed reference image. This reduces decoding latency and increases decoding speed.

[0022] In another possible implementation, the display image between the last bitstream segment of the n bitstream segments of the reference image and the reference enabled first image corresponding to the currently acquired reconstructed reference image cannot reference the knowledge image.

[0023] Update the decoded image buffer

[0024] Let curLayerId be equal to the LayerId of the current image. After completing the image header decoding and reference image queue construction for the current image, the decoded image buffer should be updated before decoding the current image. If the current image is a knowledge image and LibraryPictureModeIndex is 2, the decoded image buffer should not be updated.

[0025] If the current image is a display image, perform the following operations:

[0026] 1) For all display images in the decoded image buffer that have the same LayerId as the current image, if the image is neither in reference image queue 0 nor reference image queue 1, then mark the image as "not referenced"; otherwise, if the image is in reference image queue 0 or reference image queue 1, mark the image as "referenced". If an image is marked as "not referenced", then subsequent images should not be marked as "referenced".

[0027] 2) For a display image in the decoded image buffer that has the same DOI as the current image, if the image is in reference image queue 0 or reference image queue 1, mark the image as "referenced"; otherwise, keep the original marking unchanged.

[0028] The following operations are performed sequentially on the decoded image buffer:

[0029] —Remove the image:

[0030] 1) If the current image is a display image, remove all display images marked as "not referenced" and "already output" from the decoded image buffer;

[0031] 2) If there is a knowledge image in the decoded image buffer with the same LayerId as the current image, and the LibraryPictureExistFlag of the current image is equal to 1, and the knowledge image index of the knowledge image referenced by the current image is different from the knowledge image index of the knowledge image in the decoded image buffer, or the current image is a knowledge image, then remove the knowledge image with the same LayerId as the current image from the decoded image buffer and set LibraryBufferEmpty[curLayerId] to 1.

[0032] —Insert image:

[0033] If LibraryPictureModeIndex is 2, LibraryBufferEmpty[curLayerId] is 1, and the LibraryPictureExistFlag of the current image is equal to 1, then the corresponding knowledge image is moved from the outside into the decoded image buffer and LibraryBufferEmpty[curLayerId] is set to 0.

[0034] Bit streams conforming to this document should meet the following requirements:

[0035] —The temporal_id of the first NumRefActive[i] reference image in the reference image queue i (i equals 0 or 1) should be less than or equal to the temporal_id of the current image.

[0036] —The LayerId of the first NumRefActive[i] reference image in the reference image queue i (i equals 0 or 1) should be less than or equal to the LayerId of the current image.

[0037] —During the decoding process, the total number of currently decoded images, "not output" displayed images, and knowledge images should not exceed the value of MaxDpbSize.

[0038] Thirdly, this application provides an encoding apparatus, comprising: a reconstructed reference image acquisition module, configured to acquire a reconstructed reference image based on the encoding operation after performing encoding operations on a reference image and multiple display images corresponding to the reference image to obtain an encoded image bitstream; an encoding module, configured to update a decoded image buffer according to the currently acquired reconstructed reference image after the bitstream transmission of the reference image included in the encoded image bitstream is completed; and to perform inter-frame predictive coding on the bitstream of the display images using the currently acquired reconstructed reference image as the reference image; wherein the encoded image bitstream includes: a bitstream of the reference image and bitstreams of multiple display images; the bitstream of the reference image precedes the bitstream of the reference start image corresponding to the reference image in the multiple display image bitstreams. The reference start image includes the image of the first reference reference image among the multiple display images.

[0039] In one possible implementation, the encoding module is specifically used to replace the reconstructed reference image stored in the decoded image buffer with the currently acquired reconstructed reference image.

[0040] In another possible implementation, the display image between the last bitstream segment of the n bitstream segments of the reference image and the reference enabled first image corresponding to the currently acquired reconstructed reference image cannot reference the knowledge image.

[0041] The third aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the third aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0042] Fourthly, this application provides a decoding apparatus, comprising: a communication module for receiving an encoded image bitstream, the encoded image bitstream including a reference image bitstream and a plurality of display image bitstreams, the reference image bitstream preceding the reference start image bitstream corresponding to the reference image in the plurality of display image bitstreams, the reference start image including the image of the first reference image among the plurality of display images; a reconstructed reference image acquisition module for performing a decoding operation on the encoded image bitstream and acquiring a reconstructed reference image based on the operation result, and updating the decoding image buffer according to the currently acquired reconstructed reference image; and a decoding module for using the currently acquired reconstructed reference image as a reference image to perform inter-frame prediction decoding on the display image bitstream.

[0043] In one possible implementation, the encoded image bitstream includes n bitstream segments of a reference image; the reconstructed reference image acquisition module is specifically used to: after receiving the n bitstream segments of the reference image, decode the n bitstream segments of the reference image to obtain the currently acquired reconstructed reference image, and update the decoded image buffer according to the currently acquired reconstructed reference image.

[0044] In another possible implementation, the encoded image bitstream includes n bitstream segments of a reference image; the reconstructed reference image acquisition module is specifically used to: receive the bitstream segments of the reference image, decode the bitstream segments of the reference image to obtain reconstructed sub-images of the bitstream segments of the reference image; when the decoding of the n bitstream segments of the reference image is completed, the reconstructed sub-images of the n bitstream segments of the reference image are stitched together to obtain the currently acquired reconstructed reference image, and the decoded image buffer is updated according to the currently acquired reconstructed reference image.

[0045] In another possible implementation, the display image between the last bitstream segment of the n bitstream segments of the reference image and the reference enabled first image corresponding to the currently acquired reconstructed reference image cannot reference the knowledge image.

[0046] The fourth aspect and any two implementations thereof correspond to the second aspect and any two implementations thereof, respectively. The technical effects of the fourth aspect and any two implementations thereof can be found in the technical effects of the second aspect and any two implementations thereof mentioned above, and will not be repeated here.

[0047] Fifthly, this application provides an encoding apparatus, comprising: one or more processors; a memory for storing one or more computer programs or instructions; wherein when the one or more computer programs or instructions are executed by the one or more processors, the one or more processors implement the method as described in any one of the first aspects.

[0048] In a sixth aspect, this application provides a decoding apparatus, comprising: one or more processors; a memory for storing one or more computer programs or instructions; wherein when the one or more computer programs or instructions are executed by the one or more processors, the one or more processors implement the method as described in any one of the second aspects.

[0049] In a seventh aspect, this application provides an encoding apparatus, including a processor for performing the method as described in any one of the first aspects.

[0050] Eighthly, this application provides a decoding apparatus, including a processor for performing the method as described in any one of the second aspects.

[0051] Ninthly, this application provides an electronic device, including: a processing circuit and an interface circuit; wherein the interface circuit is used to couple with a memory external to an encoding / decoding device and to provide a communication interface for the processing circuit to access the memory; the processing circuit is used to execute program instructions in the memory to implement the method as described in any of the first to second aspects.

[0052] In practical implementation, the electronic device can be a chip, the input circuit can be an input pin, the output circuit can be an output pin, and the processing circuit can be a transistor, gate circuit, flip-flop, and various logic circuits. The input signal received by the input circuit can be received and input by, for example, but not limited to, a receiver, and the signal output by the output circuit can be output to, for example, but not limited to, a transmitter and transmitted by the transmitter. Furthermore, the input circuit and the output circuit can be the same circuit, which is used as the input circuit and the output circuit at different times. This application does not limit the specific implementation of the processor and various circuits.

[0053] In one implementation, the electronic device can be a wireless communication device, i.e., a computer device that supports wireless communication functionality. Specifically, the wireless communication device can be a terminal such as a smartphone, or a wireless access network device such as a base station. A system-on-a-chip (SoC) is also called a system-on-chip (SoC). Communication chips may include baseband processing chips and radio frequency (RF) processing chips. The baseband processing chip is sometimes referred to as a modem or baseband chip. The RF processing chip is sometimes referred to as an RF transceiver or RF chip. In physical implementation, some or all of the communication chips can be integrated within the SoC chip. For example, the baseband processing chip is integrated into the SoC chip, while the RF processing chip is not integrated with the SoC chip. The interface circuit can be the RF processing chip in the wireless communication device, and the processing circuit can be the baseband processing chip in the wireless communication device.

[0054] In another implementation, the electronic device can be a component of a wireless communication device, such as an integrated circuit product like a system-on-a-chip (SoC) or communication chip. The interface circuit can be an input / output interface, interface circuit, output circuit, input circuit, pins, or related circuits on the chip or chip system. The processor can also be represented as a processing circuit or logic circuit.

[0055] In a tenth aspect, this application provides a computer-readable storage medium storing program code, which, when executed by a processor, implements the method described in any one of the first to second aspects.

[0056] Eleventhly, this application provides a chip comprising: at least one processor. The at least one processor is configured to perform the method as described in any one of the first to second aspects.

[0057] Optionally, the chip also includes memory. At least one processor is used to execute code in the memory, and when the at least one processor executes the code, the chip implements the method as described in any one of the first to second aspects.

[0058] Alternatively, the chip described above can also be an integrated circuit.

[0059] In a twelfth aspect, this application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the method as described in any one of the first to second aspects. Attached Figure Description

[0060] Figure 1 is a schematic diagram of a video sequence provided in this application;

[0061] Figure 2 is an exemplary block diagram of a decoding system provided in this application;

[0062] Figure 3 is a flowchart illustrating an encoding method provided in this application;

[0063] Figure 4 is a flowchart illustrating a decoding method provided in this application;

[0064] Figure 5 is a schematic diagram of an encoding and decoding process provided in this application;

[0065] Figure 6 is a schematic diagram of another encoding and decoding process provided in this application;

[0066] Figure 7 is a schematic diagram of another encoding and decoding process provided in this application;

[0067] Figure 8 is a schematic diagram of another encoding and decoding process provided in this application;

[0068] Figure 9 is a schematic diagram of another encoding and decoding process provided in this application;

[0069] Figure 10 is a schematic diagram of a decoding device provided in this application;

[0070] Figure 11 is a schematic diagram of the structure of a decoder provided in this application. Detailed Implementation

[0071] To facilitate understanding, the main terms used in this application will be explained first.

[0072] Video encoding compression refers to compressing the original video signal using a specific compression algorithm. The compressed video stream (e.g., Moving Picture Experts Group 4 (MP4) file) can be stored and transmitted more conveniently.

[0073] Reconstructed image: This refers to the lossy image obtained after decoding an encoded image bitstream. The reconstructed image can be displayed and played back. Both the encoding and decoding ends can include the same reconstructed image.

[0074] At the encoding end, the reconstructed image can be used as a reference image for the image to be encoded to perform inter-frame predictive coding.

[0075] At the decoding end, the reconstructed image can be used as a reference image for the bitstream of the image to be decoded to perform inter-frame prediction decoding.

[0076] Random Access Point (RAP): In video coding, a random access point is a bitstream of encoded images that includes at least one RAP. RAPs are typically intra-coded pictures (I-frames). I-frames use only information within the current frame for predictive coding to eliminate spatial redundancy, without relying on other frames.

[0077] When random access occurs, the image bitstream before the RAP can be skipped, and decoding of the subsequent image bitstream can begin from that RAP, playing the reconstructed display image from the decoded output. From a video playback perspective, random access can be understood as dragging the player's progress bar to achieve random video jumps when watching on-demand videos (or local playback videos); or, when packet loss and a black screen occur while watching a live stream, the client will access the video stream and continue playing the video when the next RAP arrives.

[0078] Random Access Segment (RAS): A RAS consists of all images from one RAP to the next. Images in a RAS are either predictive (P) frames or bi-directional interpolated prediction (B) frames. P-frames or B-frames require temporal prediction using already encoded images to eliminate temporal redundancy.

[0079] For example, please refer to Figure 1, which is a schematic diagram of a video sequence provided in this application. Figure 1 shows 11 video frames a1 to a11. Among them, a2, a6, and a10 are I-frames and serve as RAPs, and all video frames between a2 and a6 are P-frames. RAS1 includes a2 to a5. RAS2 includes a6 to a9. As can be seen from Figure 1, each P-frame needs to be encoded with reference to the previous m video frames, that is, inter-frame prediction is performed with reference to one or more previous frames, where m is an integer greater than 1. For example, a3 can be encoded with reference to a2, and a4 can be encoded with reference to a2 and / or a3.

[0080] Cross RAP Reference (CRR) coding: When encoding video frames in a RAS, video frames in other RASs can be referenced for encoding.

[0081] In most videos (such as movies or television programs), many scenes appear alternately, and scenes displayed at different times may be similar. The interval between two adjacent similar scenes is usually relatively long. Therefore, two adjacent similar scenes may be in the same RAP but with a long interval, or their temporal correlation may span across RAPs. For example, video standards generally specify the length of the RAS (Rapid Availability) as 1 second or 2 seconds, and the interval between two adjacent similar scenes is usually close to the length of the RAS (e.g., 0.7 seconds) or greater than the length of the RAS (e.g., 3 seconds). Furthermore, in video scenarios such as security surveillance, similar scenes often appear for a longer period of time, so there may also be two similar scenes in the same RAP but with a long interval, or their temporal correlation may span across RAPs.

[0082] CRR Images: The entire image uses intra-frame predictive coding, and its reconstructed image can be stored in the decoded image buffer for a long time, serving as a reference image for multiple RAS images. CRR images are divided into displayable and non-displayable CRR images. The bitstream of a non-displayable CRR image can be interleaved with the bitstream of a display image during transmission, while the bitstream of a displayable CRR image cannot be interleaved with the bitstream of a display image during transmission. In the SVAC standard, CRR images are also called knowledge images. For example, as shown in Figure 1, a9 and a11 are CRR images.

[0083] RL image: The entire image can be coded using inter-frame prediction and intra-frame prediction, and the reference image is only one CRR image. RL image refers to an image that only references the CRR image.

[0084] Display images: I images, P images, B images, RL images, etc., that can be displayed in the encoded sequence, excluding CRR images.

[0085] Sequence Parameter Set (SPS): Contains a set of global parameters for a video coding sequence, such as color space, width and height of each frame, and coding level.

[0086] Image Parameter Set (PPS): Contains coding parameters for one or more images, such as quantization matrices and filtering parameters.

[0087] Image header: Contains encoded parameters for an image, such as frame number.

[0088] Image coding data: contains predictions, residuals, and other data that encode the content of an image.

[0089] Header information: refers to all encoding parameters used to encode and decode an image. SPS, PPS, and PH are all header information.

[0090] Network Abstraction Layer Unit (NALU): A syntax structure containing NALU header information and payload information. The NALU header contains an indication of the type (NUT) of the subsequent payload data and other relevant information. The payload information data appears in RBSP format, which also includes scattered anti-spoofing bytes. The bitstream encoded by the encoder is encapsulated in NALU format. In this paper, a NALU with NUT A is also referred to as an A NALU.

[0091] To address the issue that the decoder can only move the reconstructed CRR image from the CRR image bitstream into the decoding image buffer when it receives the first CRR image after receiving the CRR image (which is also a reference image), resulting in high system complexity and the inability to update the decoding image buffer according to the bitstream order, this application provides an encoding / decoding method. This method receives an encoded image bitstream, which includes a reference image bitstream and multiple display image bitstreams. It then performs decoding on the encoded image bitstream. Upon acquiring a reconstructed reference image, the decoding image buffer is updated based on the currently acquired reconstructed reference image. The currently acquired reconstructed reference image is then used as the reference image for inter-frame prediction decoding of the display image bitstream. The reference image bitstream precedes the reference start image bitstream corresponding to the reference image among the multiple display image bitstreams. The reference start image includes the image of the first reference reference image among the multiple display images.

[0092] Therefore, when the decoder performs inter-frame prediction decoding, it updates the decoded image buffer based on the reconstructed reference image it has acquired. It does not need to wait for the first image that only references the CRR image after receiving the CRR image before it can move the reconstructed CRR image of the CRR image bitstream into the decoded image buffer. The decoded image buffer can be updated in the order of the bitstream, which reduces the complexity of the system layer.

[0093] The implementation of the encoding / decoding method provided in this application will now be described in detail with reference to the accompanying drawings.

[0094] Figure 2 is a schematic diagram of the structure of a decoding system provided in this application, such as a video decoding system 10 (or simply decoding system 10) that can utilize the technology of this application. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) in the video decoding system 10 represent devices, etc., that can be used to perform various technologies according to the various examples described in this application.

[0095] As shown in Figure 2, the decoding system 10 includes a source device 12, which provides encoded image data 21, such as encoded images, to a destination device 14 for decoding the encoded image data 21.

[0096] The source device 12 includes an encoder 20, and optionally may include an image source 16, a preprocessor (or preprocessing unit) 18 such as an image preprocessor, and a communication interface (or communication unit) 22.

[0097] Image source 16 may include or may be any type of image capture device for capturing real-world images, and / or any type of image generation device, such as a computer graphics processor for generating computer animation images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). Image source may be any type of memory or storage device storing any of the images described above.

[0098] To distinguish the processing performed by the preprocessor (or preprocessing unit) 18, the image (or image data) 17 may also be referred to as the raw image (or raw image data) 17.

[0099] The preprocessor 18 receives the raw image data 17 and preprocesses it to obtain a preprocessed image (or preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It is understood that the preprocessing unit 18 may be an optional component.

[0100] The video encoder (or encoder) 20 is used to receive preprocessed image data 19 and provide encoded image data 21.

[0101] The communication interface 22 in the source device 12 can be used to: receive encoded image data 21 and send encoded image data 21 (or other arbitrarily processed version) to another device such as the destination device 14 or any other device via the communication channel 13 for storage or direct reconstruction.

[0102] The source device 12 may also include a memory (not shown in FIG2) that can be used to store at least one of the following data: raw image data 17, preprocessed image (or preprocessed image data) 19, and encoded image data 21.

[0103] The target device 14 includes a decoder 30, and optionally may include a communication interface (or communication unit) 28, a post-processor (or post-processing unit) 32 and a display device 34.

[0104] The communication interface 28 in the destination device 14 is used to receive encoded image data 21 (or other processed versions) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is an encoded image data storage device, and to provide the encoded image data 21 to the decoder 30.

[0105] Communication interfaces 22 and 28 can be used to send or receive encoded image data (or encoded data 21) through a direct communication link between source device 12 and destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0106] For example, the communication interface 22 can be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission encoding or processing, so as to transmit it on a communication link or communication network.

[0107] Communication interface 28 corresponds to communication interface 22. For example, it can be used to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded image data 21.

[0108] Both communication interface 22 and communication interface 28 can be configured as a one-way communication interface or a two-way communication interface as indicated by the arrow pointing from the source device 12 to the destination device 14 in FIG2, and can be used to send and receive messages, etc., to establish a connection, acknowledge and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.

[0109] The video decoder (or decoder) 30 is used to receive encoded image data 21 and provide decoded image data (or decoded image data) 31.

[0110] The post-processor 32 is used to post-process the decoded image data 31 (also called reconstructed image data) to obtain post-processed image data 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color adjustment, trimming or resampling, or any other processing to generate the decoded image data 31 for display by the display device 34, etc.

[0111] Display device 34 is used to receive post-processed image data 33 to display the image to a user or viewer. Display device 34 can be or includes any type of display for representing the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0112] The target device 14 may also include a memory (not shown in FIG2), which may be used to store at least one of the following data: encoded image data 21, decoded image data 31 and post-processed image data 33.

[0113] The decoding system 10 also includes a training engine 25, which is used to train the encoder 20 to process the input image or image region or image block to obtain a feature map of the input image or image region or image block, and to obtain an estimated probability distribution of the feature map and encode the feature map according to the estimated probability distribution.

[0114] The training engine 25 is also used to train the decoder 30 to obtain the estimated probability distribution of the bitstream, decode the bitstream according to the estimated probability distribution to obtain the feature map, and decode the feature map to reconstruct the reconstructed image.

[0115] Although Figure 2 illustrates source device 12 and destination device 14 as independent devices, device embodiments may also include both source device 12 and destination device 14, or the functions of both source device 12 and destination device 14 simultaneously; that is, they may include both source device 12 or its corresponding functions and destination device 14 or its corresponding functions. In these embodiments, source device 12 or its corresponding functions and destination device 14 or its corresponding functions may be implemented using the same hardware and / or software, or through separate hardware and / or software, or any combination thereof.

[0116] As described, the presence and (accurate) division of different units or functions in the source device 12 and / or destination device 14 shown in Figure 2 may vary depending on the actual device and application, which is obvious to those skilled in the art.

[0117] Next, the implementation methods of the encoding and decoding methods provided in this application will be described.

[0118] Figure 3 is a flowchart illustrating an encoding method provided in this application. This method can be applied to the encoder shown in Figure 2. As shown in Figure 3, the method includes the following steps.

[0119] Step 310: After performing encoding operations on the CRR image and multiple display images corresponding to the CRR image to obtain the encoded image bitstream, the reconstructed CRR image is obtained based on the encoding operation.

[0120] For example, step 310 above may specifically include the following processes 401 to 403 and optional implementation methods.

[0121] 401. Obtain the CRR images corresponding to multiple display images.

[0122] Multiple display images can be consecutive display images within a video sequence. The encoder can be set with a preset duration, dividing the video sequence into at least one video segment according to the preset duration, and acquiring a CRR image corresponding to each video segment. A video segment can include multiple display images. The preset duration can be longer than the current RAS length, for example, the preset duration can be 6s, 7s, or 10s. Correspondingly, multiple display images include multiple RAPs, and each RAP can be a P-frame or a B-frame.

[0123] A reconstructed CRR image can serve as a reference image for multiple subsequent display images to be encoded. Optionally, any one of the multiple display images can be used as the CRR image corresponding to the multiple display images. Alternatively, multiple CRR images corresponding to the multiple display images can be constructed based on the multiple display images. This application does not limit the method of obtaining the CRR image.

[0124] 402. Encode the CRR image and multiple display images to obtain the encoded image bitstream.

[0125] The encoded image bitstream includes a CRR image bitstream and multiple display image bitstreams. In some embodiments, the encoded image bitstream includes n bitstream segments of the CRR image. The n bitstream segments of the CRR image are interleaved with the multiple display image bitstreams. The encoded image bitstream includes n interleaved CRR image bitstream segments and multiple display image bitstreams, where n is an integer greater than 1.

[0126] The bitstream of the CRR image precedes the bitstream of the reference enabled first image corresponding to the CRR image in the multiple display image bitstreams. The reference enabled first image includes the image of the first reference CRR image among the multiple display images.

[0127] The encoded image bitstream is a single-channel bitstream. When n CRR image bitstream segments are transmitted, the transmission order (i.e. the encoder output order) is always before the bitstream of the first display image that references that CRR image.

[0128] The CRR image is used to generate the reconstructed image, which can then serve as a reference image for inter-frame predictive coding of the display image to be encoded. The CRR image does not need to be displayed. After encoding a display image in a video sequence, a display image bitstream and a reconstructed display image are obtained. Therefore, in this embodiment, the reconstructed image includes two types: a reconstructed display image and a reconstructed CRR image.

[0129] The reference images for the displayed images include: the reconstructed displayed image and / or the currently active reconstructed CRR image. The currently active reconstructed CRR image can be stored in the encoder's buffer space, such as in the encoder's decoded picture buffer (DPB). The currently active reconstructed CRR image refers to the reconstructed CRR image that is in effect at the moment of encoding operations on each of the multiple displayed images.

[0130] When encoding each of the multiple display images, at least one display image has already been encoded, and correspondingly, at least one reconstructed display image exists. Optionally, any number of reconstructed display images from all currently obtained reconstructed display images can be used as reference images for that display image. For example, the reference images for the display image include at least one reconstructed display image obtained from the encoding operation performed on the most recently output display image.

[0131] For example, when encoding the third display image out of multiple display images, the current encoding yields the first display image bitstream and the first reconstructed display image corresponding to the first display image, and the second display image bitstream and the second reconstructed display image corresponding to the second display image. Therefore, when encoding the third display image out of multiple display images, the first and second reconstructed display images can be used as reference images for the third display image.

[0132] The currently active reconstructed CRR image is obtained through encoding operations performed on the CRR image. Optionally, it can be obtained when encoding a group of CRR image bitstream segments. The group of CRR image bitstream segments can include: the latest n output CRR image bitstream segments, or the latest n output CRR image bitstream segments corresponding to the same CRR image. Here, "latest output" refers to the latest output at the moment the encoding operation on the display image is performed.

[0133] During the encoding process of the entire video sequence, there is no currently active reconstructed CRR image before the output of the nth CRR image bitstream segment. Therefore, for the display image bitstream output before the nth CRR image bitstream segment, the reference image used during encoding only includes the reconstructed display image.

[0134] Since n CRR image bitstream segments and multiple display image bitstreams are interleaved into one encoded image bitstream, the encoded image bitstream can be transmitted using existing transmission protocols and stored using existing file formats.

[0135] 403. Encode the preset syntax identifiers into the encoded image bitstream.

[0136] The preset syntax identifier includes at least one of the following: quantity information, image information, and position information. The quantity information indicates n, the image information indicates the target image, the bitstream of the target image is transmitted after n CRR image bitstream segments, and the position information indicates the position of the n CRR image bitstreams in the encoded image bitstream.

[0137] Encoding the preset syntax identifier into the encoded image bitstream can be done by directly adding the preset syntax identifier to the encoded image bitstream, or by adding it to the encoded image bitstream after encoding and compression. This application does not limit this.

[0138] Preset syntax identifiers are used by the decoder to determine if CRR slices are missing. These identifiers can be located in higher-level parameters of the encoded image bitstream, including at least one of the following: SPS, PPS, and image header. SPS includes a set of global parameters for the video coding sequence, such as color space, the size (width and height) of each frame, and coding level. PPS includes coding parameters for at least one frame, such as quantization matrix and filtering parameters. The image header includes coding parameters for a single frame, such as frame number.

[0139] For example, for SPS, there can be one SPS before every N CRR image bitstream segments, or one SPS before the entire bitstream. For PPS, there can be one PPS before every multiple bitstreams. For image headers, there can be an image header before the first CRR image bitstream segment, and a slice header or segment header before each of the other non-first CRR image bitstream segments.

[0140] Since the decoder parses higher-level parameters after receiving the bitstream, placing the preset syntax identifier within these parameters allows the decoder to quickly determine if any missing CRR image bitstream segments exist. This enables the decoder to respond promptly to transmission failures, enhancing the robustness of the encoding and decoding process.

[0141] The target image can be: the first display image that uses a CRR image as a reference image (the image header of this display image has a CRR image enable first image flag), the first display image that uses only a CRR image as a reference image, a display image whose bitstream is transmitted after n CRR image bitstream segments, or a CRR image whose bitstream is transmitted after n CRR image bitstream segments. Before transmitting the bitstream of the target image, the transmission of n CRR image bitstream segments needs to be completed. For example, assuming the target image is a CRR image, the bitstream segments of the previously acquired CRR image and the bitstream segments of the target image should not be interleaved during transmission; that is, the transmission of the bitstream segments of the target image should not be initiated before all the bitstream segments of the previously acquired CRR image have been encoded and transmitted. In other words, the target image can be used as a reference enable first image.

[0142] Image information may include: the frame number of the target image, the decoding sequence number of the bit stream of the target image, the difference between the frame numbers of the target image and the CRR image, or the difference between the decoding sequence numbers of the bit stream of the target image and the bit stream of the CRR image. This application does not limit this.

[0143] Location information can include a location identifier for each CRR image bitstream segment. The location identifier of a CRR image bitstream segment indicates where the next CRR image bitstream segment will be transmitted. The location identifier of a CRR image bitstream segment can be: a frame sequence number (referring to the frame sequence number of a display image bitstream preceding or following the CRR image bitstream segment), a decoding sequence number (referring to the decoding sequence number of a display image bitstream preceding or following the CRR image bitstream segment), the difference between the frame sequence number and the frame sequence number corresponding to the next CRR image bitstream segment, or the difference between the decoding sequence number and the decoding sequence number corresponding to the next CRR image bitstream segment, etc., as long as it indicates where the CRR image bitstream segment will be transmitted.

[0144] Step 320: After the transmission of multiple bitstream segments of the CRR image included in the encoded image bitstream is completed, update the decoded image buffer according to the currently acquired reconstructed CRR image.

[0145] The reference image used for inter-frame predictive coding is stored in a designated buffer. For example, the encoder and decoder each have their own designated buffer, which may be, for example, a decoded picture buffer (DPB).

[0146] After the encoder transmits the n CRR image bitstream segments, which include the encoded image bitstream, to the decoder, the specified buffer is updated using the currently acquired reconstructed CRR image.

[0147] For example, the reconstructed reference image stored in the decoded image buffer can be replaced with the currently acquired reconstructed reference image. Specifically, this replacement could be done by overwriting the reconstructed reference image stored in the decoded image buffer, or by removing the reconstructed reference image from the decoded image buffer and moving the currently acquired reconstructed CRR image into the decoded image buffer.

[0148] In this embodiment, the reference image used for inter-frame predictive coding is stored in a designated buffer. The reference image can then be determined by updating the designated buffer using the currently acquired reconstructed CRR image, without the need for additional marking and recording information, making it more efficient.

[0149] In some embodiments, the encoder encodes n sub-images of the CRR image to obtain n bitstream segments of the CRR image. After sending the n bitstream segments of the CRR image, the currently acquired reconstructed CRR image is moved into the decoded image buffer.

[0150] In other embodiments, the encoder encodes a sub-image of the CRR image to obtain a bitstream segment of the CRR image, and transmits the bitstream segment of the CRR image. When the encoder receives the last bitstream segment of the transmitted CRR image, it concatenates the reconstructed sub-images of n bitstream segments of the CRR image to obtain the currently acquired reconstructed reference image, and updates the decoded image buffer based on the currently acquired reconstructed reference image.

[0151] It should be noted that the displayed image between the last bitstream segment of the n bitstream segments of the reference image and the reference enabled first image corresponding to the currently acquired reconstructed reference image cannot be a reference knowledge image.

[0152] For example, after all CRR sub-images in a CRR image have been transmitted, subsequent encoded images may not immediately use the reconstructed CRR image reference, nor may they reference the previous CRR image. However, they can be encoded as display images of non-RL images and reference the images preceding them.

[0153] Optionally, after the encoder transmits the n CRR image bitstream segments included in the encoded image bitstream to the decoder, the reconstructed CRR image reference must be used immediately, that is, the encoded image after the last CRR sub-image bitstream is transmitted is the RL image.

[0154] Optionally, before the encoder completes the transmission of the n CRR image bitstream segments included in the encoded image bitstream to the decoder, the previous reconstructed CRR image stored in the specified buffer can be used as a reference image.

[0155] Step 330: Using the currently acquired reconstructed CRR image as a reference image, perform inter-frame predictive coding on the displayed image.

[0156] The image information is used to indicate the reference enabled first image, which includes the first image in the displayed image that is referenced to the reconstructed CRR image.

[0157] Optionally, the preset syntax identifier may also include bitstream segment information, which is used to indicate the display image for each CRR image bitstream segment used as a reference image.

[0158] Optionally, after obtaining the reconstructed CRR image based on the encoding operation, the encoding method provided in this application embodiment may further include:

[0159] When the currently processed display image is a non-reference enabled first image of the currently acquired reconstructed CRR image, an image other than the reconstructed CRR image used in the previous processed display image is used as the reference image, and inter-frame predictive coding is performed on the display image.

[0160] Since the image in the decoded image buffer has been replaced with the currently acquired reconstructed CRR image, and the reconstructed CRR image used by the previously processed display image has been removed, the image other than the reconstructed CRR image used by the previously processed display image is used as the reference image to perform inter-frame predictive coding on the display image. This ensures that the reference image can be accurately determined for long-distance multiple display images in cross-RAP scenes, further improving the coding success rate and avoiding coding failure.

[0161] When the currently processed display image is the reference image for the currently acquired reconstructed CRR image, the currently acquired reconstructed CRR image is used as the reference image to perform inter-frame predictive coding on the display image.

[0162] In this embodiment of the application, when the encoder performs inter-frame predictive coding, after the transmission of n CRR image bitstream segments is completed, the reconstructed CRR images of the n CRR image bitstream segments are stored in the updated decoded image buffer, so that the decoded image buffer can be updated in the order of the bitstream, reducing the complexity of the system layer.

[0163] Please refer to Figure 4, which is a flowchart illustrating a decoding method provided in this application. This method can be applied to the decoder shown in Figure 2. As shown in Figure 4, the method includes the following steps.

[0164] Step 410: Receive the encoded image bitstream.

[0165] The encoded image bitstream includes a bitstream of a CRR image and a bitstream of multiple display images. In some embodiments, the encoded image bitstream includes n bitstream segments of the CRR image. The n bitstream segments of the CRR image are interleaved with the multiple display image bitstreams. The encoded image bitstream includes the interleaved CRR image bitstream segments, the multiple display image bitstreams, quantity information, and image information. The image information is used to indicate a reference enabled first image. The reference enabled first image includes the first image in the display images that references the reconstructed CRR image. The multiple bitstream segments of the CRR image precede the bitstream of the reference enabled first image corresponding to the CRR image in the multiple display image bitstreams.

[0166] Step 420: Decode the encoded image bitstream and, based on the operation result, obtain the reconstructed CRR image. Update the decoded image buffer according to the currently obtained reconstructed reference image.

[0167] Steps 410 and 420 above may specifically include the processes 501 to 505 below, as well as optional examples.

[0168] 501. Receive encoded image bitstream, which includes interleaved CRR image bitstream segments and display image bitstream.

[0169] 502. Decode the encoded image bitstream.

[0170] Each time a display image bitstream is decoded, a reconstructed display image is obtained. The reference images for the display image bitstream include the reconstructed display image and / or the currently active reconstructed CRR image. The currently active reconstructed CRR image can be stored in the decoder's buffer space, such as in the decoder's DPB.

[0171] The reference image used when decoding any display image bitstream in the encoded image bitstream is the same as the reference image used by the encoder when encoding the display image bitstream. Each display image bitstream may carry a reference image identifier, and the decoder determines the reference image of the display image bitstream based on the reference image identifier it carries when decoding the display image bitstream.

[0172] The reconstructed display image is obtained by decoding the previous display image bitstream; a display image bitstream is decoded to obtain a reconstructed display image.

[0173] The currently active reconstructed CRR image is obtained by decoding the previous CRR image bitstream segments. Optionally, the currently active reconstructed CRR image can be obtained by decoding a group of CRR image bitstream segments. The group of CRR image bitstream segments includes: the first n CRR image bitstream segments of the display image bitstream, or the first n CRR image bitstream segments of the display image bitstream corresponding to the same CRR image. During the decoding process of the encoded image bitstream corresponding to the entire video sequence, there is no currently active reconstructed CRR image before the decoding of the received nth CRR image bitstream segment is completed. Therefore, for the display image bitstream before the nth CRR image bitstream segment, the reference image used during decoding only includes the reconstructed display image.

[0174] 503. Obtain the preset syntax identifier from the encoded image bitstream.

[0175] The default syntax identifier includes at least one of the following: quantity information, image information, and position information. Quantity information indicates n. Image information indicates the target image transmitted after n CRR image bitstream segments in the encoded image bitstream. Position information indicates the position of the n CRR image bitstream segments within the encoded image bitstream.

[0176] When the preset syntax identifier only includes quantity information, the decoder parses the quantity information (indicating n) from the encoded image bitstream upon receiving the first CRR image bitstream segment. When the preset syntax identifier only includes image information, the decoder parses the image information (indicating the target image) from the encoded image bitstream upon receiving the first CRR image bitstream segment; all n CRR image bitstream segments must be received before the target image bitstream. When the preset syntax identifier only includes position information, the decoder parses the position information from the encoded image bitstream upon receiving the first CRR image bitstream segment.

[0177] 504. Based on the preset syntax identifier, determine whether there are n CRR image bitstream segments in the encoded image bitstream, where n CRR image bitstream segments correspond to one CRR image, and n is an integer greater than 1.

[0178] In some embodiments, after parsing the quantity information from the encoded image bitstream, the decoder starts counting from the first CRR image bitstream segment received from the encoded image bitstream. That is, the counter starts counting when the first CRR image bitstream segment is received. Then, the display image bitstream and CRR image bitstream segments are received, and the counter value increases by 1 for each received CRR image bitstream segment.

[0179] Upon receiving the target bitstream, if the number of received CRR image bitstream segments is n (i.e., the counter value is n), then it is determined that there are n CRR image bitstream segments in the encoded image bitstream, and the CRR image bitstream reception is successful. If the number of received CRR image bitstream segments is less than n, then it is determined that there are no n CRR image bitstream segments in the encoded image bitstream, and the CRR image bitstream reception fails.

[0180] The target bitstream is the display image bitstream and the reference image is a CRR image, or the preset syntax identifier also includes image information, and the target bitstream is the bitstream of the target image indicated by the image information.

[0181] The decoder can, upon successful reception of the CRR image bitstream segment corresponding to the previous CRR image, default to the next received CRR image bitstream segment as the first CRR image bitstream segment. Alternatively, if only the first CRR image bitstream segment in the encoded image bitstream has an image header, the decoder will use the CRR image bitstream segment after the image header as the first CRR image bitstream segment. Or, if the CRR image bitstream segments corresponding to a CRR image are transmitted sequentially, the decoder can use the CRR image bitstream segment with the smallest sequence number as the first CRR image bitstream segment.

[0182] In other embodiments, after parsing image information (indicating the target image) from the encoded image bitstream, the decoder decodes the encoded image bitstream and determines the size of the reconstructed CRR image upon receiving the target image bitstream. The reconstructed CRR image is obtained by decoding the received CRR image bitstream segments. When the size of the reconstructed CRR image is equal to the size of the CRR image, it is determined that there are n CRR image bitstream segments in the encoded image bitstream, and the CRR image bitstream reception is successful. When the size of the reconstructed CRR image is smaller than the size of the CRR image, it is determined that there are not n CRR image bitstream segments in the encoded image bitstream, and the CRR image bitstream reception fails.

[0183] In other embodiments, after the decoder parses the position information from the encoded image bitstream, if a CRR image bitstream segment is received at each position indicated by the position information, it determines that there are n CRR image bitstream segments in the encoded image bitstream, and the CRR image bitstream reception is successful. If no CRR image bitstream segment is received at at least one position indicated by the position information, it determines that there are no n CRR image bitstream segments in the encoded image bitstream, and the CRR image bitstream reception fails.

[0184] 505. If there are n CRR image bitstream segments in the encoded image bitstream, then obtain the reconstructed reference image and update the decoded image buffer based on the currently obtained reconstructed reference image.

[0185] If there are n CRR image bitstream segments in the encoded image bitstream, the CRR image bitstream is successfully received, and the specified buffer is updated using the currently acquired reconstructed CRR image.

[0186] For example, the reconstructed reference image stored in the decoded image buffer can be replaced with the currently acquired reconstructed reference image. Specifically, this replacement could be done by overwriting the reconstructed reference image stored in the decoded image buffer, or by removing the reconstructed reference image from the decoded image buffer and moving the currently acquired reconstructed CRR image into the decoded image buffer.

[0187] In some embodiments, after the decoder receives n bitstream segments of the CRR image, it buffers the bitstream segments of the CRR image. That is, after collecting n bitstream segments of the CRR image, it decodes the n bitstream segments of the CRR image to obtain the currently acquired reconstructed CRR image and moves the currently acquired reconstructed CRR image into the decoded image buffer.

[0188] In other embodiments, after receiving a bitstream segment of the CRR image, the decoder decodes the bitstream segment to obtain a reconstructed sub-image of the CRR image bitstream segment, and buffers the reconstructed sub-image of the CRR image bitstream segment. When the decoder receives the last bitstream segment of the CRR image, after decoding the last bitstream segment of the CRR image, the decoding of n bitstream segments of the CRR image is completed. The reconstructed sub-images of the n bitstream segments of the CRR image are stitched together to obtain the currently acquired reconstructed reference image, and the decoded image buffer is updated according to the currently acquired reconstructed reference image.

[0189] Step 430: Using the currently acquired reconstructed reference image as the reference image, perform inter-frame prediction decoding on the bitstream of the display image, and continue to decode the bitstream of the encoded image to obtain the reconstructed display image.

[0190] If n CRR image bitstream segments are missing from the encoded image bitstream, the decoder can determine that the CRR image bitstream reception is incorrect, and transmission error handling can be performed. Transmission error handling includes: discarding or skipping the display image bitstream composed of the images decoded from the missing CRR image bitstream segments of the reference image, and requesting the encoder to retransmit the missing CRR image bitstream segments, etc., which are not limited in this embodiment.

[0191] If there are n CRR image bitstream segments in the encoded image bitstream, the decoder can determine that the CRR image bitstream has been successfully received. It then updates the specified buffer with the currently acquired reconstructed CRR image and uses the currently acquired reconstructed reference image as the reference image to perform inter-frame prediction decoding on the display image bitstream. When the decoder receives the first image among multiple display images that references the reference image (i.e., the RL image), it performs decoding operations with reference to the currently acquired reconstructed CRR image to obtain the reconstructed display image.

[0192] It should be noted that the displayed image between the last bitstream segment of the n bitstream segments of the CRR image and the reference enabled first image corresponding to the currently acquired reconstructed CRR image cannot be a reference knowledge image.

[0193] For example, after all CRR sub-images in a CRR image have been transmitted, subsequent encoded images do not need to immediately use the reconstructed CRR image reference, nor can they reference the previous CRR image. However, they can be encoded as non-RL image display images and reference the images preceding them. In other words, if an encoded image bitstream is between the last fragment bitstream of a CRR image and the first encoded image (RL image) that references that CRR image, then that encoded image cannot reference the previous CRR image for encoding and decoding, nor can it reference the currently acquired reconstructed CRR image.

[0194] Optionally, it can also be constrained that there are no other coded image bitstreams between the last segment bitstream of the CRR image and the first coded image bitstream that references the CRR image. That is, after the decoder receives the n CRR image bitstream segments included in the coded image bitstream and completes their transmission, it must immediately use the reconstructed CRR image reference, i.e., the coded image after the last CRR sub-image bitstream has been transmitted is the RL image.

[0195] In summary, the decoding method provided in this application performs a decoding operation on the received encoded image bitstream. The encoded image bitstream includes interleaved CRR image bitstream segments and a display image bitstream. A preset syntax identifier is obtained from the encoded image bitstream. Based on the preset syntax identifier of the encoded image bitstream, it is determined whether there are n CRR image bitstream segments in the encoded image bitstream. The n CRR image bitstream segments correspond to one CRR image. If there are n CRR image bitstream segments in the encoded image bitstream, the decoding operation on the encoded image bitstream continues to obtain a reconstructed display image. The reference image of the display image bitstream includes: the reconstructed display image and / or the currently effective reconstructed CRR image. The reconstructed display image is obtained by decoding the previous display image bitstream, and the currently effective reconstructed CRR image is obtained by decoding the previous CRR image bitstream segments. The currently active reconstructed CRR image can serve as a reference image for the current display image bitstream to be decoded. This reconstructed CRR image is obtained based on CRR image bitstream sub-segments, which are obtained at the encoding end based on the CRR image. Since the CRR image corresponds to multiple display images, and these display images can include multiple RAPs, cross-RAP referencing is possible. Furthermore, the n CRR image bitstream segments and multiple display image bitstreams are interleaved into a single bitstream and transmitted to the decoder. In this approach, all processes can be completed within the decoder, eliminating the need for system-level cooperation. This achieves cross-RAP referencing while simultaneously decoupling the decoder from the system layer.

[0196] Furthermore, based on the preset syntax identifier, it can determine whether there are missing CRR image bitstream segments, enabling the encoding / decoding layer and the system layer to cooperate better, thereby allowing the decoder to react to transmission faults in a timely manner and enhancing the robustness of the encoding / decoding process.

[0197] In addition, when the decoder performs inter-frame prediction decoding, it updates the decoded image buffer based on the reconstructed reference image as soon as it obtains the reconstructed reference image. It does not need to wait for the first image that only references the CRR image after receiving the CRR image before it can move the reconstructed CRR image of the CRR image bitstream into the decoded image buffer. The decoded image buffer can be updated in the order of the bitstream, which reduces the complexity of the system layer.

[0198] For example, please refer to Figure 5, which is a schematic diagram of an encoding and decoding process provided in this application. The encoder encodes the CRR image and multiple display images to obtain an encoded image bitstream and sends the encoded image bitstream to the decoder. Figure 5 illustrates this using three CRR image bitstream segments b1 to b3 (n=3) and five display image bitstreams c1 to c5 as examples. The decoder decodes the bitstreams in the encoded image bitstream sequentially according to their order, obtaining five reconstructed display images c1′ to c5′ and a reconstructed CRR image. The five reconstructed display images are used for display, while the reconstructed CRR image is only used as a reference image when decoding the display image bitstream and is not used for display.

[0199] In one implementation, the CRR image can be first segmented into n CRR sub-images, and then the CRR sub-images can be encoded, including the following process:

[0200] 601. Obtain the CRR images corresponding to multiple display images.

[0201] This process can refer to the aforementioned process 401, and will not be repeated here in the embodiments of this application.

[0202] 602. Divide the CRR image into n CRR sub-images, where n is an integer greater than 1.

[0203] 603. Interweave n CRR sub-images among multiple display images to obtain the image sequence to be encoded.

[0204] In the sequence of images to be encoded, the last image is the display image, meaning that all CRR images are located before the last display image.

[0205] Optionally, one or more CRR sub-images can be interspersed between every two display images, as long as all n CRR sub-images are interleaved between multiple display images. This application embodiment does not limit the interleaving method.

[0206] 604. Perform encoding operations on the CRR sub-images and the display image in the image sequence to be encoded in sequence to obtain the encoded image bitstream.

[0207] The order of the displayed image bitstream in the encoded image bitstream is the same as the order of the corresponding displayed images in the image sequence to be encoded. The order of the CRR sub-image bitstream in the encoded image bitstream is the same as the order of the corresponding CRR sub-images in the image sequence to be encoded.

[0208] In this implementation, there are several encoding examples for generating the currently active reconstructed CRR image. These examples are explained below.

[0209] Encoding Example 1: After encoding each CRR sub-image, a CRR image bitstream segment and a reconstructed CRR sub-image are obtained. The encoder caches each reconstructed CRR sub-image, for example, in the encoder's DPB. When n reconstructed CRR sub-images are cached, they are concatenated to obtain the currently active reconstructed CRR image. If an active reconstructed CRR image exists before concatenating the n reconstructed CRR sub-images, the previously active reconstructed CRR image is removed from the cache when concatenating the n reconstructed CRR sub-images to obtain the currently active reconstructed CRR image. In this example, the CRR image bitstream segment group includes: the latest n output CRR image bitstream segments corresponding to the same CRR image.

[0210] Optionally, the first display image selected as the currently active reconstructed CRR image can be marked as the enabled first image of the currently active reconstructed CRR image.

[0211] Encoding Example 2, based on Encoding Example 1, when encoding the first CRR sub-image in the image sequence to be encoded begins, if a currently active reconstructed CRR image exists (referring to the reconstructed CRR image in effect at the moment encoding the first CRR sub-image in the image sequence to be encoded begins), then the currently active reconstructed CRR image is removed (removed from the cache space). When encoding a target display image among multiple display images, the reference image for the target display image includes the reconstructed display image, which includes the display image between the first and last CRR sub-images in the image sequence to be encoded. Since there is no active reconstructed CRR image when encoding the target display image, the reference image for the target display image only includes the reconstructed display image.

[0212] In this encoding example, when encoding the first CRR sub-image to the last CRR sub-image in the sequence of images to be encoded, there is no need to cache the previous effective reconstructed CRR image, saving the cache space of one frame of images during this period.

[0213] Encoding Example 3: When encoding a CRR sub-image to obtain a reconstructed CRR sub-image, if a currently active reconstructed CRR image exists, the target region in the currently active reconstructed CRR image (referring to the reconstructed CRR image in effect at the moment the CRR sub-image is encoded to obtain the reconstructed CRR sub-image) is updated to the obtained reconstructed CRR sub-image, thus obtaining the currently active reconstructed CRR image. The positional relationship between the target region and the currently active reconstructed CRR image is the same as the target positional relationship, which is the positional relationship between the cached reconstructed CRR sub-image and the corresponding CRR image. In this example, the CRR image bitstream segment group includes: the latest n output CRR image bitstream segments.

[0214] In this encoding example, after obtaining a reconstructed CRR sub-image, the reconstructed CRR sub-image is not cached. Instead, the target region in the currently active reconstructed CRR image is directly replaced with the reconstructed CRR sub-image, which can save the cache occupied by the reconstructed CRR sub-image.

[0215] Encoding Example 4: The encoder buffers each output CRR image bitstream segment (e.g., buffered in the encoder's DPB). When n CRR image bitstream segments are buffered, each of the n buffered CRR image bitstream segments is decoded and reconstructed to obtain n reconstructed CRR sub-images. The n reconstructed CRR sub-images are then concatenated to obtain the currently active reconstructed CRR image. That is, the encoder's encoding operation for multiple display images and CRR images also includes the decoding and reconstruction of n CRR image bitstream segments. If an active reconstructed CRR image exists before the CRR image bitstream is decoded and reconstructed, the previously active reconstructed CRR image is removed from the buffer space when the CRR image bitstream is decoded and reconstructed to obtain the currently active reconstructed CRR image. In this example, the CRR image bitstream segment group includes: the latest n output CRR image bitstream segments corresponding to the same CRR image.

[0216] Optionally, the first image selected as the currently active reconstructed CRR image can be used as the display image for reference, and marked as the enabled first image of the currently active reconstructed CRR image, i.e., the reference enabled first image.

[0217] In this encoding example, CRR image bitstream segments are directly cached, and decoding and reconstruction are performed only after caching n CRR image bitstream segments to obtain the reconstructed CRR image, without needing to cache each reconstructed CRR sub-image. Since the cache space occupied by the CRR image bitstream segments is smaller than the cache space occupied by the reconstructed CRR sub-images, certain cache space can be saved.

[0218] In another implementation, the CRR image can be encoded first to obtain CRR image bitstream segments, and then the CRR image bitstream segments can be segmented, which may include the following process:

[0219] 701. Obtain the CRR images corresponding to multiple display images.

[0220] This process can refer to the aforementioned process 401, and will not be repeated here in the embodiments of this application.

[0221] 702. Encode the CRR image to obtain the CRR image bitstream.

[0222] 703. Segment the CRR image bitstream to obtain n CRR image bitstream segments, where n is an integer greater than 1.

[0223] 704. Encode multiple display images sequentially to obtain multiple display image bitstreams.

[0224] 705. Interweave n CRR image bitstream segments between multiple display image bitstreams to obtain an encoded image bitstream.

[0225] In the encoded image bitstream, the last image bitstream is the display image bitstream, meaning that all CRR image bitstreams are located before the last display image bitstream.

[0226] Optionally, one or more CRR image bitstream segments can be inserted between every two display image bitstreams, as long as all n CRR image bitstream segments are interleaved in multiple display image bitstreams. This application does not limit the interleaving method.

[0227] It should be noted that processes 704 and 705 can be executed simultaneously. The encoder can predetermine the order of each CRR image bitstream segment in the subsequent encoded image bitstream and determine the currently active reconstructed CRR image based on this order. Only when n CRR image bitstream segments are output can the reconstructed CRR image obtained during the encoding operation of the CRR image be used as the currently active reconstructed CRR image. That is, in the encoded image bitstream, when encoding the display image after the nth CRR image bitstream segment, the reconstructed CRR image obtained during the encoding operation of the CRR image can be used as a reference image for inter-frame predictive coding.

[0228] In this implementation, there are several examples of generating the currently active reconstructed CRR image. These examples are explained below.

[0229] In encoding example 5, after encoding the CRR image, the encoder obtains the target reconstructed CRR image and the CRR image bitstream. The encoder buffers the target reconstructed CRR image. When outputting the last CRR image bitstream segment out of n CRR image bitstream segments, the buffered target reconstructed CRR image is used as the currently effective reconstructed CRR image. In this example, the CRR image bitstream segment group includes the latest n CRR image bitstream segments corresponding to the same CRR image.

[0230] If an active reconstructed CRR image exists before the last CRR image bitstream segment in the output of n CRR image bitstream segments, then when the cached target reconstructed CRR image is used as the currently active reconstructed CRR image, the previously active reconstructed CRR image is removed from the cache space. Furthermore, in this case, an additional frame buffer is required to store the target reconstructed CRR image.

[0231] Optionally, the first display image selected as the currently active reconstructed CRR image can be marked as the enabled first image of the currently active reconstructed CRR image.

[0232] Encoding Example 6: The encoder can buffer the CRR image bitstream. When outputting the last CRR image bitstream segment out of n CRR image bitstream segments, the buffered CRR image bitstream is decoded and reconstructed to obtain the currently effective reconstructed CRR image. In this example, the CRR image bitstream segment group includes: the latest n CRR image bitstream segments corresponding to the same CRR image.

[0233] If a reconstructed CRR image exists before the cached CRR image bitstream is decoded and reconstructed, the previously active reconstructed CRR image is removed from the cache when the cached CRR image bitstream is decoded and reconstructed to obtain the currently active reconstructed CRR image. Furthermore, in this case, an additional cache is required to store the CRR image bitstream.

[0234] In this encoding example, the entire CRR image bitstream is stored before the last CRR image bitstream segment in the output of n CRR image bitstream segments, instead of storing the entire reconstructed CRR image. Since the buffer space occupied by the CRR image bitstream is smaller than that occupied by the reconstructed CRR image, some buffer space can be saved.

[0235] It should be noted that in the above encoding examples, after the encoder has transmitted the n CRR image bitstream segments included in the encoded image bitstream to the decoder, the last effective reconstructed CRR image is removed from the decoded image buffer, and the decoded image buffer is updated using the currently acquired reconstructed CRR image.

[0236] For different encoding examples, there are also different decoding examples on the decoding end. The following describes various decoding examples.

[0237] In decoding example 1, the decoder performs decoding operations sequentially according to the order of each bitstream in the encoded image bitstream. After decoding each received CRR image bitstream segment, it obtains and buffers the reconstructed CRR sub-image. If, according to the preset syntax identifier, there are n CRR image bitstream segments in the encoded image bitstream, then when there are n reconstructed CRR sub-images in the buffer, the n reconstructed CRR sub-images are concatenated to obtain the currently effective reconstructed CRR image. If there is an effective reconstructed CRR image before concatenating the n reconstructed CRR sub-images, then when concatenating the n reconstructed CRR sub-images to obtain the currently effective reconstructed CRR image, the previously effective reconstructed CRR image is removed from the buffer space. In this example, the CRR image bitstream segment group includes: the first n CRR image bitstream segments of the display image bitstream corresponding to the same CRR image.

[0238] The process of determining whether there are missing CRR image bitstream segments can be achieved through the quantity information in the preset syntax identifier, as described in the aforementioned embodiments. The embodiments in this application will not be repeated here.

[0239] Optionally, the first display image bitstream that selects the currently active reconstructed CRR image as the reference image can be marked as the enabled first image bitstream of the currently active reconstructed CRR image.

[0240] Decoding Example 2, based on Decoding Example 1, when decoding the first CRR image bitstream segment in the encoded image bitstream, if a currently active reconstructed CRR image exists (referring to the reconstructed CRR image in effect at the moment the decoding operation begins on the first CRR image bitstream segment in the encoded image bitstream), then the currently active reconstructed CRR image is removed (removed from the buffer space). When decoding the target display image bitstream in multiple display image bitstreams, the reference image of the target display image bitstream includes the reconstructed display image. The target display image bitstream includes the display image bitstream between the first and last CRR image bitstream segments in the encoded image bitstream. Since there is no active reconstructed CRR image when encoding the target display image bitstream, the reference image of the target display image bitstream only includes the reconstructed display image.

[0241] In this decoding example, when decoding the first CRR image bitstream segment to the last CRR image bitstream segment in the encoded image bitstream, there is no need to cache the previous effective reconstructed CRR image, saving the cache space of one frame of image during this period.

[0242] In decoding example 3, the decoder performs decoding operations sequentially according to the order of each bitstream in the encoded image bitstream. After decoding the received CRR image bitstream segments, a reconstructed CRR sub-image is obtained. If there are no lost CRR image bitstream segments at the current point (i.e., there are n CRR image bitstream segments in the encoded image bitstream), and a currently effective reconstructed CRR image exists, the target region in the currently effective reconstructed CRR image is updated to the obtained reconstructed CRR sub-image, resulting in the currently effective reconstructed CRR image. The positional relationship between the target region and the currently effective reconstructed CRR image is the same as the target positional relationship, which is the positional relationship between the cached reconstructed CRR sub-image and the corresponding CRR image. In this example, the CRR image bitstream segment group includes: the first n CRR image bitstream segments of the displayed image bitstream.

[0243] The process of determining whether there are missing CRR image bitstream segments can be achieved through the position information in the preset syntax identifier, as described in the aforementioned embodiments. The embodiments in this application will not be repeated here.

[0244] In this decoding example, after obtaining a reconstructed CRR sub-image, the reconstructed CRR sub-image is not cached. Instead, the target region in the currently active reconstructed CRR image is directly replaced with the reconstructed CRR sub-image, which can save the cache occupied by the reconstructed CRR sub-image.

[0245] In decoding example 4, when the decoder receives the first n-1 CRR image bitstream segments, it does not perform decoding immediately but buffers each received CRR image bitstream segment. Subsequently, if there are n CRR image bitstream segments in the encoded image bitstream, the reconstructed CRR image is used as the currently active reconstructed CRR image. The reconstructed CRR image is obtained by decoding the buffered n CRR image bitstream segments separately to obtain n reconstructed CRR sub-images, and then concatenating these n reconstructed CRR sub-images. In this example, the CRR image bitstream segment group includes: the first n CRR image bitstream segments of the display image bitstream corresponding to the same CRR image.

[0246] Optionally, the first display image bitstream that selects the currently active reconstructed CRR image as the reference image can be marked as the enabled first image bitstream of the currently active reconstructed CRR image.

[0247] The process of determining whether there are missing CRR image bitstream segments can be achieved through quantity information and / or image information in the preset syntax identifier, as described in the foregoing embodiments. The embodiments in this application will not be repeated here.

[0248] In decoding example 5, when the decoder receives the first n-1 CRR image bitstream segments, it does not perform decoding immediately but buffers each received CRR image bitstream segment. If there are n CRR image bitstream segments in the encoded image bitstream, the reconstructed CRR image is used as the currently active reconstructed CRR image. The reconstructed CRR image is obtained by concatenating the buffered n CRR image bitstream segments, and then decoding the concatenated CRR image bitstream. If there is an active reconstructed CRR image before decoding the CRR image bitstream, the previously active reconstructed CRR image is removed from the buffer when decoding the CRR image bitstream to obtain the currently active reconstructed CRR image. In this example, the CRR image bitstream segment group includes: the first n CRR image bitstream segments of the displayed image bitstream corresponding to the same CRR image.

[0249] The process of determining whether there are missing CRR image bitstream segments can be achieved through quantity information and / or image information in the preset syntax identifier, as described in the foregoing embodiments. The embodiments in this application will not be repeated here.

[0250] Optionally, the first display image bitstream that selects the currently effective reconstructed CRR image as the reference image can be marked as the enabled first image bitstream of the currently effective reconstructed CRR image, i.e., the reference enabled first image bitstream.

[0251] In decoding examples 4 and 5, CRR image bitstream segments are directly cached, and decoding is performed only after caching n CRR image bitstream segments to obtain the reconstructed CRR image, without needing to cache each reconstructed CRR sub-image. Since the cache space occupied by the CRR image bitstream segment is smaller than the cache space occupied by the reconstructed CRR sub-image, certain cache space can be saved.

[0252] It should be noted that if there are n CRR image bitstream segments in the encoded image bitstream, the CRR image bitstream is successfully received, and the specified buffer is updated using the currently acquired reconstructed CRR image.

[0253] In the first possible implementation, after the decoder receives n bitstream segments of the CRR image, it decodes the n bitstream segments of the CRR image to obtain the currently acquired reconstructed CRR image, and moves the currently acquired reconstructed CRR image into the decoded image buffer.

[0254] In the second possible implementation, after the decoder receives a bitstream segment of the CRR image, it decodes the bitstream segment to obtain a reconstructed sub-image of the CRR image's bitstream segment. When the decoder receives the last bitstream segment of the CRR image, it decodes the last bitstream segment, thus completing the decoding of all n bitstream segments of the CRR image. The reconstructed sub-images of the n bitstream segments of the CRR image are then stitched together to obtain the currently acquired reconstructed reference image, and the decoded image buffer is updated based on the currently acquired reconstructed reference image.

[0255] The encoding and decoding process will be illustrated below with reference to the accompanying drawings. In the following encoding and decoding process, five display images d1 to d5 and one corresponding CRR image e will be used as examples, and it is assumed that the number of sub-images of the CRR image is n = 3.

[0256] I. Taking the first possible implementation as an example, after receiving n bitstream segments of the CRR image, the n bitstream segments of the CRR image are decoded to obtain the currently acquired reconstructed CRR image.

[0257] For example, please refer to Figure 6, which is a schematic diagram of an encoding and decoding process provided in this application. The CRR image e is divided into three CRR sub-images e1 to e3. e1 is interspersed between d1 and d2, e2 is interspersed between d2 and d3, and e3 is interspersed between d3 and d4 to obtain the image sequence to be encoded. Then, the image sequence to be encoded is encoded. First, d1 is encoded. Since no new CRR image bitstream segment has been output at this time, the buffer space only stores the first reconstructed CRR image; the currently effective reconstructed CRR image is the first reconstructed CRR image. A reference image is selected to encode d1 to obtain the display image bitstream d1′ and the reconstructed display image (not shown in Figure 6). e1 is encoded to obtain the CRR image bitstream segment e1′ and the reconstructed CRR sub-image e1″, and e1″ is buffered.

[0258] Encoding operation is performed on d2. Since the three reconstructed CRR sub-images have not yet been cached, the currently effective reconstructed CRR image is still the first reconstructed CRR image. The reference image is selected to encode d2 to obtain the display image bitstream d2′ and the reconstructed display image (not shown in Figure 6). Encoding operation is performed on e2 to obtain the CRR image bitstream segment e2′ and the reconstructed CRR sub-image e2″, and e2″ is cached.

[0259] Encoding operation is performed on d3. Since the three reconstructed CRR sub-images have not yet been cached, the currently effective reconstructed CRR image is still the first reconstructed CRR image. The reference image is selected to encode d3 to obtain the display image bitstream d3′ and the reconstructed display image (not shown in Figure 6). Encoding operation is performed on e3 to obtain the CRR image bitstream segment e3′ and the reconstructed CRR sub-image e3″, and e3″ is cached.

[0260] At this point, the buffer space contains three reconstructed CRR sub-images e1″ to e3″. These are concatenated to obtain the second reconstructed CRR image e″. The first reconstructed CRR image is removed from the decoded image buffer, and the second reconstructed CRR image e″ is moved from the buffer space into the decoded image buffer. The currently active reconstructed CRR image is e″.

[0261] Encode d4 by selecting the first reconstructed CRR image and the reference image other than e″ to encode d4 to obtain the display image bitstream d4′ and the reconstructed display image (not shown in Figure 6).

[0262] Encode d5, and the currently effective reconstructed CRR image is e″. Select the currently effective reconstructed CRR image e″ and encode d5 to obtain the display image bitstream d5′ and the reconstructed display image (not shown in Figure 6). At this time, the encoded image bitstream is obtained.

[0263] The decoder performs decoding operations sequentially according to the order of the bitstreams in the encoded image bitstream. First, it decodes d1′. Since no new CRR image bitstream segment has been decoded yet, the decoded image buffer only stores the first reconstructed CRR image; the currently active reconstructed CRR image is the first one. A reference image is selected to decode d1′, resulting in the reconstructed display image d1″. The reference image selected for decoding d1′ is the same as the reference image selected for encoding d1.

[0264] Upon receiving e1′, buffer e1′, e1′ being the first CRR image bitstream segment. Parse the quantity information (indicating 3) from the encoded image bitstream and start counting to 1.

[0265] Decoding d2′ is performed. Since the three reconstructed CRR sub-images are not yet cached, the currently active reconstructed CRR image is still the first reconstructed CRR image. A reference image is selected to decode d2′ to obtain the reconstructed display image d2″. The reference image selected for decoding d2′ is the same as the reference image selected for encoding d2.

[0266] Upon receiving e2′, buffer e2′ and increment the counter by 1 to get 2.

[0267] Decoding d3′ is performed. Since the three reconstructed CRR sub-images are not yet cached, the currently active reconstructed CRR image is still the first reconstructed CRR image. A reference image is selected to decode d3′ to obtain the reconstructed display image d3″. The reference image selected for decoding d3′ is the same as the reference image selected for encoding d3.

[0268] Upon receiving e3′, buffer e3′ and increment the counter by 1 to get 3.

[0269] Since the last CRR image bitstream segment e3′ of the CRR image is received, the decoding operation is performed on e1′, e2′ and e3′ to obtain e1″, e2″ and e3″. The reconstructed CRR image e″ is obtained by splicing e1″, e2″ and e3″. The first reconstructed CRR image is removed from the decoded image buffer and the reconstructed CRR image e″ is moved into the decoded image buffer.

[0270] Subsequently, d4′ was received. d4′ did not refer to the reconstructed CRR image e″ corresponding to e1″ to e3″. At this time, the number of CRR image bitstream segments received was 3. There are 3 CRR image bitstream segments in the encoded image bitstream. The CRR image bitstream was successfully received.

[0271] Decoding operation is performed on d4′. Since the first reconstructed CRR image in the decoded image buffer has been removed, the first reconstructed CRR image cannot be referenced. d4′ is not an RL image, and there is no currently effective reconstructed CRR image. The first reconstructed CRR image and a reference image other than e″ are selected to decode d4′ to obtain the reconstructed display image d4″. The reference image selected for decoding d4′ is the same as the reference image selected for encoding d4′.

[0272] Decode d5′ and the currently effective reconstructed CRR image is e″. Select the reference image as the currently effective reconstructed CRR image e″ and decode d5′ to obtain the reconstructed display image d5″. The reference image selected for decoding d5′ is the same as the reference image selected for encoding d5′.

[0273] Second, taking the second possible implementation as an example, after receiving a bitstream segment of the CRR image, the bitstream segment of the CRR image is decoded, and the reconstructed sub-image of n bitstream segments of the CRR image is spliced ​​together to obtain the currently acquired reconstructed CRR image.

[0274] For example, please refer to Figure 7, which is a schematic diagram of an encoding and decoding process provided in this application. The encoding and decoding process shown in Figure 7 is similar to that in Figure 6. The difference between the decoder side and Figure 6 is that: e1′ is received and decoded, and e1″ is buffered; e2′ is received and decoded, and e2″ is buffered; e3′ is received and decoded, and e3″ is buffered. Since the last CRR image bitstream segment e3′ of the CRR image is received and decoded, e1″, e2″, and e3″ are obtained. e1″, e2″, and e3″ are concatenated to obtain the reconstructed CRR image e″, and the reconstructed CRR image e″ is moved into the decoding image buffer. Since the first reconstructed CRR image in the decoding image buffer has been removed when d4′ is decoded, the first reconstructed CRR image cannot be referenced, and there is no currently effective reconstructed CRR image. Therefore, the first reconstructed CRR image and the reference image other than e″ are selected to decode d4′ to obtain the reconstructed display image d4″.

[0275] Other processes can be referred to the relevant description in Figure 6, and will not be repeated here in the embodiments of this application.

[0276] During the encoding and decoding process shown in Figures 6 and 7 above, after receiving the last CRR image bitstream segment of the CRR image, there may still be a non-RL image bitstream between the last CRR image bitstream segment and the enabled first image bitstream of the CRR image. This constrains the image to not refer to the previous CRR image, nor to the currently acquired reconstructed CRR image.

[0277] In other embodiments, after receiving the last CRR image bitstream segment of the CRR image, the next CRR image bitstream segment is followed by the CRR image enable first image bitstream, constraining the RL image to reference the currently acquired reconstructed CRR image. This is illustrated in Figures 8 and 9 below.

[0278] Third, taking the first possible implementation as an example, after receiving n bitstream segments of the CRR image, the n bitstream segments of the CRR image are decoded to obtain the currently acquired reconstructed CRR image.

[0279] For example, please refer to Figure 8, which is a schematic diagram of an encoding and decoding process provided in this application. The encoding and decoding process shown in Figure 8 is similar to that in Figure 6. The difference between the decoder and Figure 6 is that after receiving the last CRR image bitstream segment of the CRR image, the encoded image after the last CRR image bitstream segment is used as the reference start image.

[0280] For example, when the reconstructed CRR image e″ is moved into the decoding image buffer and the decoding operation is performed on d4′, the first reconstructed CRR image in the decoding image buffer has been removed, and the first reconstructed CRR image cannot be referenced. The currently effective reconstructed CRR image is e″, so e″ is selected as the reference image to decode d4′ and obtain the reconstructed display image d4″. When decoding d5′, e″ is selected as the reference image to decode d5′ and obtain the reconstructed display image d5″. Optionally, when decoding d5′, e″ can be chosen as the reference image instead of e″, and another image can be selected to decode d5′ and obtain the reconstructed display image d5″.

[0281] Fourth, taking the second possible implementation as an example, after receiving a bitstream segment of the CRR image, the bitstream segment of the CRR image is decoded, and the reconstructed sub-image of n bitstream segments of the CRR image is spliced ​​together to obtain the currently acquired reconstructed CRR image.

[0282] For example, please refer to Figure 9, which is a schematic diagram of an encoding and decoding process provided in this application. The encoding and decoding process shown in Figure 9 is similar to that in Figure 7. The difference between the decoder and Figure 7 is that after receiving the last CRR image bitstream segment of the CRR image, the encoded image after the last CRR image bitstream segment is used as the reference start image.

[0283] For example, when the reconstructed CRR image e″ is moved into the decoding image buffer, and the decoding operation is performed on d4′, the first reconstructed CRR image in the decoding image buffer has been removed, so it is impossible to refer to the first reconstructed CRR image. The currently effective reconstructed CRR image is e″, so e″ is selected as the reconstructed CRR image to decode d4′ and obtain the reconstructed display image d4″. When decoding d5′, e″ is selected as the reconstructed CRR image to decode d5′ and obtain the reconstructed display image d5″.

[0284] It is understood that, in order to achieve the functions in the above embodiments, the encoder includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and method steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0285] The encoding and decoding method provided according to this embodiment has been described in detail above with reference to Figures 1 to 9. The decoding device and decoder provided according to this embodiment will be described below with reference to Figures 10 and 11.

[0286] Figure 10 is a schematic diagram of a possible decoding device provided in this embodiment. These decoding devices can be used to implement the decoding function in the above method embodiments, and therefore can also achieve the beneficial effects of the above method embodiments. In this embodiment, the decoding device can be the decoder 30 shown in Figure 2, or it can be a module (such as a chip) applied to a terminal device or server.

[0287] As shown in Figure 10, the decoding device 1000 includes a communication module 1010, a reconstructed reference image acquisition module 1020, a decoding module 1030, and a storage module 1040. The decoding device 1000 is used to implement the function of the decoder in the method embodiment shown in Figure 4 above.

[0288] The communication module 1010 is used to receive an encoded image bit stream, which includes multiple bit stream segments of an interleaved reference image and multiple display image bit streams. The multiple bit stream segments of the reference image precede the bit stream of the reference enabled first image corresponding to the reference image in the multiple display image bit streams. The reference enabled first image includes the image of the first reference reference image among the multiple display images.

[0289] The reconstruction reference image acquisition module 1020 is used to perform decoding operations on the encoded image bitstream, acquire a reconstruction reference image based on the operation results, and update the decoding image buffer according to the currently acquired reconstruction reference image.

[0290] The decoding module 1030 is used to perform inter-frame prediction decoding on the bitstream of the displayed image by using the currently acquired reconstructed reference image as a reference image.

[0291] Storage module 1040 is used to store the reconstructed reference image.

[0292] It should be understood that the decoding device 1000 in this embodiment can be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. Alternatively, when the decoding method shown in FIG4 is implemented in software, the decoding device 1000 and its various modules can also be software modules.

[0293] A more detailed description of the communication module 1010, the reconstructed reference image acquisition module 1020, the decoding module 1030, and the storage module 1040 can be obtained directly from the relevant descriptions in the method embodiment shown in Figure 4, and will not be repeated here.

[0294] Figure 11 is a schematic diagram of the structure of a decoder 1100 provided in this embodiment. As shown in Figure 11, the decoder 1100 includes a processor 1110, a bus 1120, a memory 1130, and a communication interface 1140.

[0295] It should be understood that in this embodiment, the processor 1110 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), ASICs, FPGAs, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0296] The processor may also be a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, or one or more integrated circuits used to control the execution of the program in this application.

[0297] The communication interface 1140 is used to enable communication between the decoder 1100 and external devices or components. In this embodiment, the communication interface 1140 is used to receive encoded image bitstreams.

[0298] Bus 1120 may include a pathway for transmitting information between the aforementioned components (such as processor 1110 and memory 1130). In addition to a data bus, bus 1120 may also include a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus 1120 in the figure.

[0299] As an example, decoder 1100 may include multiple processors. A processor may be a multi-CPU processor. Here, "processor" can refer to one or more devices, circuits, and / or computing units for processing data (e.g., computer program instructions). Processor 1110 may perform decoding operations on the encoded image bitstream, and based on the operation results, acquire a reconstructed reference image, update the decoded image buffer according to the currently acquired reconstructed reference image, and use the currently acquired reconstructed reference image as a reference image to perform inter-frame prediction decoding on the bitstream of the displayed image.

[0300] It is worth noting that Figure 11 only shows the decoder 1100 as including one processor 1110 and one memory 1130. Here, the processor 1110 and the memory 1130 are used to indicate a type of device or equipment. In specific embodiments, the number of each type of device or equipment can be determined according to business needs.

[0301] The memory 1130 can correspond to the storage medium used to store information such as reconstructed reference images in the above method embodiments, such as a disk, like a mechanical hard disk or a solid-state hard disk.

[0302] The decoder 1100 described above can be a general-purpose device or a dedicated device. For example, the decoder 1100 can be an x86 or ARM-based server, or other dedicated servers, such as a policy control and charging (PCC) server. This application does not limit the type of decoder 1100.

[0303] It should be understood that the decoder 1100 according to this embodiment can correspond to the decoding device 1000 in this embodiment, and can correspond to the corresponding subject that executes any method in FIG4. The above and other operations and / or functions of each module in the decoding device 1000 are respectively for implementing the corresponding processes of each method in FIG4. For the sake of brevity, they will not be described in detail here.

[0304] This application also provides a system including an encoder and a decoder as shown in FIG11. The encoder and decoder are used to implement the method steps shown in FIG3 and FIG4 above, which will not be described in detail here for the sake of brevity.

[0305] The foregoing primarily describes the encoding and decoding methods provided in this application from the perspective of the device. It is understood that, in order to achieve the above functions, the device includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0306] This application embodiment can divide the device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0307] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to execute any of the methods described in the embodiments of this application.

[0308] This application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be executed by a computer or a device with encoding / decoding capabilities, using computer programs or instructions to control related hardware. The computer program or set of instructions can be stored in the aforementioned computer-readable storage medium. When executed, the computer program or set of instructions can include the processes described in the above method embodiments. The computer-readable storage medium can be an internal storage unit of the control terminal in any of the foregoing embodiments, such as a hard disk or memory of the control terminal. The aforementioned computer-readable storage medium can also be an external storage device of the control terminal, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the control terminal. Further, the aforementioned computer-readable storage medium can include both the internal storage unit of the control terminal and external storage devices. The aforementioned computer-readable storage medium is used to store the aforementioned computer program or instructions, as well as other programs and data required by the control terminal. The aforementioned computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0309] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0310] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0311] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0312] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0313] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0314] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0315] To make the objectives, technical solutions, and advantages of this application clearer, the above solutions, in conjunction with the accompanying drawings, provide a clear and complete description of the technical solutions in this application. Obviously, the described embodiments are only a portion of the embodiments of this application, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0316] The terms "first," "second," etc., used in the specification, embodiments, claims, and drawings of this application are for distinguishing purposes only and should not be construed as indicating or implying relative importance or order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0317] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0318] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A decoding method, characterized in that, include: Receive an encoded image bitstream, the encoded image bitstream including a reference image bitstream and a plurality of display images bitstreams, the reference image bitstream preceding the reference enabled first image bitstream corresponding to the reference image in the plurality of display image bitstreams, the reference enabled first image including the first image in the plurality of display images that references the reference image; The encoded image bitstream is decoded, and a reconstructed reference image is obtained based on the operation result. The decoded image buffer is then updated according to the currently obtained reconstructed reference image. Using the currently acquired reconstructed reference image as a reference image, inter-frame prediction decoding is performed on the bitstream of the displayed image.

2. The method according to claim 1, characterized in that, The encoded image bitstream includes n bitstream segments of the reference image; The process of decoding the encoded image bitstream, obtaining a reconstructed reference image based on the result, and updating the decoded image buffer according to the currently obtained reconstructed reference image includes: After receiving n bitstream segments of the reference image, the n bitstream segments of the reference image are decoded to obtain the currently acquired reconstructed reference image, and the decoded image buffer is updated according to the currently acquired reconstructed reference image.

3. The method according to claim 1, characterized in that, The encoded image bitstream includes n bitstream segments of the reference image; The process of decoding the encoded image bitstream, obtaining a reconstructed reference image based on the result, and updating the decoded image buffer according to the currently obtained reconstructed reference image includes: Receive a bitstream segment of the reference image, decode the bitstream segment of the reference image, and obtain a reconstructed sub-image of the bitstream segment of the reference image; Once the decoding of n bitstream segments of the reference image is completed, the reconstructed sub-images of the n bitstream segments of the reference image are stitched together to obtain the currently acquired reconstructed reference image, and the decoded image buffer is updated according to the currently acquired reconstructed reference image.

4. The method according to any one of claims 1-3, characterized in that, The encoded image bitstream includes n bitstream segments of the reference image; the display image between the last bitstream segment of the reference image and the reference enabled first image corresponding to the currently acquired reconstructed reference image cannot be a reference knowledge image.

5. An encoding method, characterized in that, include: After encoding the reference image and multiple display images corresponding to the reference image to obtain an encoded image bitstream, the reconstructed reference image is obtained based on the encoding operation. After the transmission of multiple bitstream segments of the reference image included in the encoded image bitstream is completed, the decoded image buffer is updated according to the currently acquired reconstructed reference image; Using the currently acquired reconstructed reference image as a reference image, inter-frame predictive coding is performed on the bitstream of the displayed image; The encoded image bitstream includes a reference image bitstream and multiple display image bitstreams; the reference image bitstream precedes the reference enabled first image bitstream corresponding to the reference image in the multiple display image bitstreams, and the reference enabled first image includes the first image in the multiple display images that references the reference image.

6. The method according to claim 5, characterized in that, Update the decoded image buffer based on the currently acquired reconstructed reference image, including: Replace the reconstructed reference image stored in the decoded image buffer with the currently acquired reconstructed reference image.

7. The method according to claim 6, characterized in that, The encoded image bitstream includes n bitstream segments of the reference image; the display image between the last bitstream segment of the reference image and the reference enabled first image corresponding to the currently acquired reconstructed reference image cannot be a reference knowledge image.

8. A decoding device, comprising at least one processor and a memory, characterized in that, The at least one processor executes a program or instructions stored in the memory to cause the decoding device to implement the method described in any one of claims 1 to 4.

9. An encoding device, comprising at least one processor and a memory, characterized in that, The at least one processor executes a program or instructions stored in the memory to cause the encoding device to implement the method described in any one of claims 5 to 7.

10. An electronic device, characterized in that, include: Processor and memory; The processor and the memory are connected; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

11. A computer-readable storage medium for storing a computer program, characterized in that, When the computer program is run on a computer or processor, it causes the computer or processor to perform the method described in any one of claims 1 to 7.

12. A computer program product, the computer program product comprising instructions, characterized in that, When the instructions are executed on a computer or processor, they cause the computer or processor to perform the method described in any one of claims 1 to 7.

13. A chip comprising at least one processor and a memory, characterized in that, The at least one processor executes a program or instructions stored in a memory to cause the chip to implement the method described in any one of claims 1 to 7.

Images