Image coding method and apparatus

Abstract

The image coding method and device provided in the application can reduce the cost overhead and power consumption of the encoder and the decoder. The method can include: reading a first sub-image of a first frame image from a first memory and storing the first sub-image in a second memory, the first memory being an off-chip memory, and the second memory being an on-chip memory; encoding a base layer of the first sub-image based on the first sub-image to obtain a code stream of the base layer of the first sub-image and a first reconstructed image; writing the first reconstructed image into the second memory; reading the first reconstructed image and the first sub-image from the second memory; and encoding a first enhancement layer of the first sub-image based on the first reconstructed image and the first sub-image to obtain a code stream of the first enhancement layer of the first sub-image and a second reconstructed image.

Description

Technical Field

[0001] This application relates to multimedia technology. More specifically, it relates to image encoding and decoding methods and apparatus in multimedia technology. Background Technology

[0002] Short-range wireless projection technology refers to a set of technologies that transmit video data generated by mobile terminals with strong processing capabilities, such as game screens rendered by graphics processing units (GPUs), wirelessly to devices with weaker processing capabilities but better display effects, such as televisions or virtual reality (VR) headsets, after encoding and compression. This technology can solve the problems of connectivity and information sharing between mobile devices and other devices.

[0003] Applications using short-range wireless projection, such as game projection and VR glasses, involve data interaction, thus requiring the lowest possible end-to-end latency. Simultaneously, the wireless channel of Wireless Fidelity (Wi-Fi) is susceptible to environmental factors such as noise and interference, leading to unstable transmission bandwidth and data loss, resulting in image quality issues. Therefore, the projection system must have strong resistance to interference from channel variations. In summary, end-to-end latency and resistance to interference from channel variations are crucial factors affecting the performance of the projection system and the user experience.

[0004] Scalable video coding (SVC) is a technique where the encoder encodes each frame of a source video into multiple image layers. These layers correspond to different qualities or resolutions, and there is a reference relationship between them. During transmission, the image layers are transmitted sequentially from lower quality / lower resolution to higher quality / higher resolution. The more image layers the decoder receives in a frame, the better the quality of the reconstructed image.

[0005] In existing layered encoding and decoding schemes, before encoding each image layer of a frame of an image, the encoder in the system on chip (or system-on-chip, SOC) needs to read the image data required for encoding (such as the original image or the reconstructed image of the image layer) from the external memory of the SOC, and / or write the image data obtained during the encoding process (such as the encoded bitstream of the image layer or the reconstructed image of the image layer) into the external memory.

[0006] However, as the number of image layers increases, the encoder needs to access the off-chip memory more frequently, and the access bandwidth also increases accordingly. This results in significant cost and power consumption for the encoder when accessing this off-chip memory. Similarly, since the decoder's execution process is the reverse of the encoder's, the decoder also suffers from similar problems. Summary of the Invention

[0007] This application provides an image encoding and decoding method and apparatus that can reduce the cost and power consumption of encoders and decoders accessing off-chip memory.

[0008] In a first aspect, this application provides an image encoding method that can be used in an image encoding apparatus. The image encoding apparatus may include a system-on-a-chip (SOC) and an external memory. The SOC may include an encoder, an on-chip memory, and a communication interface. The encoder, the on-chip memory, and the communication interface are coupled to each other. The external memory is located outside the SOC and is coupled to the communication interface. The method may include: the encoder reading a first sub-image of a first frame image from a first memory and storing the first sub-image in a second memory, wherein the first memory is an off-chip memory and the second memory is an on-chip memory, the first frame image includes multiple sub-images, and the multiple sub-images include the first sub-image; the encoder reading the first sub-image from the on-chip memory; the encoder encoding the base layer of the first sub-image based on the first sub-image to obtain a bitstream of the base layer of the first sub-image and a first reconstructed image, wherein the first reconstructed image is a reconstructed image of the base layer of the first sub-image; the encoder writing the first reconstructed image into the on-chip memory; the encoder reading the first reconstructed image and the first sub-image from the on-chip memory; the encoder encoding a first enhancement layer of the first sub-image based on the first reconstructed image and the first sub-image to obtain a bitstream of the first enhancement layer of the first sub-image and a second reconstructed image, wherein the second reconstructed image is a reconstructed image of the first enhancement layer of the first sub-image, and the image quality of the enhancement layer is greater than the image quality of the base layer.

[0009] Optionally, the first memory can be dynamic random access memory (DRAM), and the second memory can be static random access memory (SRAM).

[0010] Optionally, the first frame image mentioned above can be any frame in the video frame sequence, and this application embodiment does not limit this.

[0011] Optionally, the first sub-image can be any one of the plurality of sub-images, and this embodiment of the application does not limit this.

[0012] Optionally, the aforementioned sub-image can be, but is not limited to, a block, macroblock (MB), or largest coding unit (LCU) in a frame image, or other block-level images. This application embodiment does not limit this. Here, a block-level image refers to an image that does not fully conform to the standard size specified by a block, macroblock, or largest coding unit, but whose size difference from the standard size of the block, macroblock, or largest coding unit is less than a preset size threshold.

[0013] In existing solutions, before encoding each image layer of the image to be encoded, the encoder needs to read the image to be encoded from the off-chip memory and store it in the on-chip memory, and then read the image to be encoded from the on-chip memory for encoding. Furthermore, for any two adjacent (i.e., adjacent image quality) image layers of the image to be encoded, the encoder writes the reconstructed image of the lower (i.e., lower image quality) image layer to the off-chip memory. Correspondingly, the encoder needs to read the reconstructed image of the lower image layer from the off-chip memory and store it in the on-chip memory, and then read the reconstructed image of the lower image layer from the on-chip memory to encode the higher (i.e., higher image quality) image layer.

[0014] In other words, the encoder in the existing solution needs to frequently access the off-chip memory to obtain the image data required for encoding and / or store the encoded image data. As the number of image layers increases, the number of reads and writes to the off-chip memory and the amount of data read and written will increase significantly. In addition, since the off-chip memory is usually characterized by slow read and write speed and high power consumption, the cost and power consumption of the encoder in the existing solution for accessing the off-chip memory and the on-chip memory are relatively large.

[0015] In this application, the encoder only needs to read the sub-image from the off-chip memory and store it in the on-chip memory once when it first obtains the sub-image to be encoded. During subsequent encoding of multiple image layers of the sub-image, it only reads the sub-image from the on-chip memory for encoding. Furthermore, for any two adjacent image layers, the encoder writes the reconstructed image of the lower (i.e., lower image quality) image layer to the on-chip memory; correspondingly, the encoder can read only the reconstructed image of the lower image layer from the on-chip memory to encode the higher (i.e., higher image quality) image layer. Therefore, the image encoding method provided in this application can reduce the number of read / write operations and the amount of data read / written by the encoder to the off-chip and on-chip memory. Moreover, since the on-chip memory typically has fast read / write speeds and low power consumption, the image encoding method provided in this application can reduce the cost and power consumption of the encoder accessing the off-chip and on-chip memory.

[0016] Optionally, the encoding method of the basic layer of the first sub-image can be intra-frame coding or inter-frame coding, and this application does not limit it.

[0017] In one possible implementation, if the first frame image is the first frame image in a video sequence, then the encoder encoding the basic layer of the first sub-image based on the first sub-image may include: encoding the basic layer of the first sub-image using intra-frame coding based on the first sub-image and an intra-frame reference image.

[0018] Optionally, this application does not limit the method of obtaining the aforementioned intra-frame reference image. For example, the intra-frame reference image can be obtained by referring to existing technologies.

[0019] In another possible implementation, if the first frame image is not the first frame image in the video sequence, the encoder may encode the basic layer of the first sub-image based on the first sub-image by using inter-frame coding based on the first sub-image and the inter-frame reference image.

[0020] Optionally, this application does not limit the method of obtaining the inter-frame reference image. For example, the inter-frame reference image can be obtained by referring to existing technologies.

[0021] In one possible implementation, the encoder can directly write the first reconstructed image into the on-chip memory. That is, the first reconstructed image is an image that has not undergone loop filtering.

[0022] In existing solutions, the reconstructed images of each image layer require in-loop filtering before being written to off-chip memory to improve image quality. However, while in-loop filtering improves image quality, it also increases the amount of data stored, requiring subsequent retrieval of the reconstructed images from this off-chip memory. This increases the encoder's bandwidth requirements for accessing off-chip memory, leading to longer image encoding times and reduced data transmission efficiency.

[0023] In this application, the reconstructed images of each image layer (such as the first reconstructed image) are directly stored in the on-chip memory without using loop filtering. This allows the first reconstructed image to be stored in the on-chip memory with a smaller data volume, and the reconstructed images of each image can be directly read from the on-chip memory for subsequent use. This reduces the encoder's bandwidth requirements for accessing external memory, thereby reducing image encoding time and improving data transmission efficiency.

[0024] In one possible implementation, the encoder can encode the enhancement layer of the first sub-image using inter-layer coding based on the first reconstructed image and the first sub-image. That is, the first reconstructed image is a reference image for inter-layer coding (from the base layer to the enhancement layer).

[0025] Optionally, after obtaining the second reconstructed image, the method further includes: the encoder writing the second reconstructed image into the second memory.

[0026] Optionally, this application does not limit the storage location of the second reconstructed image in the on-chip memory.

[0027] In one possible implementation, the on-chip memory may include storage space a and storage space b, which do not overlap. During the layered encoding process of the same sub-image, storage space a is used to store the first sub-image, and storage space b is used to store the reconstructed images of each image layer in a multiplexed manner. This allows more data to be stored in a fixed amount of storage space, thereby improving the utilization rate of the on-chip memory and reducing its cost.

[0028] In another possible implementation, the on-chip memory may include multiple storage spaces (e.g., storage space a, storage space b, and storage space c), which do not overlap. During the hierarchical encoding of the same sub-image, different storage spaces are used to store different images. This reduces the complexity of data reading and writing, thereby improving data transmission efficiency.

[0029] Optionally, the above steps are only described using the encoding of the basic layer and the encoding of the first enhancement layer of the first sub-image as examples, but this application is not limited thereto. Optionally, the image encoding device can also use a similar method to encode other image layers (image layers with higher image quality), and this application embodiment does not limit this.

[0030] In one possible implementation, the encoder can read the second reconstructed image and the first sub-image from the second memory; based on the second reconstructed image and the first sub-image, it encodes the second enhancement layer of the first sub-image to obtain the bitstream of the second enhancement layer of the first sub-image and a third reconstructed image, wherein the third reconstructed image is the reconstructed image of the second enhancement layer of the first sub-image, and the image quality of the second enhancement layer is higher than that of the first enhancement layer; the third reconstructed image is written to the second memory, and the third reconstructed image covers the second reconstructed image.

[0031] In other words, the reconstructed images of each image layer reuse a fixed storage space in the on-chip memory. This allows more data to be stored in a fixed storage space, thereby improving the utilization rate of the on-chip memory and reducing its cost.

[0032] The above description allows for the encoding of multiple image layers (such as the base layer, the first enhancement layer, and the second enhancement layer) of the first sub-image. Furthermore, the encoder can use a similar method to encode multiple image layers of other sub-images within these multiple sub-images, which will not be elaborated upon here.

[0033] In one possible implementation, if the plurality of sub-images further includes a second sub-image, and the encoding order of the second sub-image and the first sub-image is adjacent, the method may further include: after completing the encoding of the plurality of image layers of the first sub-image, encoding the plurality of image layers of the second sub-image, the plurality of image layers including the base layer and the first enhancement layer.

[0034] In other words, the layered encoding of multiple sub-images is performed serially. This allows a single encoder to sequentially complete the layered encoding of all sub-images, thus reducing encoder costs.

[0035] In another possible implementation, if the plurality of sub-images also includes a second sub-image, and the encoding order of the second sub-image and the first sub-image is adjacent, the method may further include: encoding the base layer of the second sub-image while encoding the first enhancement layer of the first sub-image.

[0036] In other words, the hierarchical encoding of multiple sub-images is performed in parallel. This requires multiple encoders (which may include multiple sub-encoders) to each complete the hierarchical encoding of one sub-image, so that the hierarchical encoding of multiple sub-images can be performed in parallel, which can reduce the encoding time.

[0037] Optionally, the first memory may include multiple storage spaces, with different storage spaces used to store bitstreams of different image layers. The method further includes: the encoder writing the bitstreams of different image layers into the storage space corresponding to each image layer in the first memory.

[0038] In one possible implementation, taking the first sub-image as having multiple image layers including a base layer and a first enhancement layer as an example, the first memory may include a first storage space and a second storage space. The first storage space is used to store the bitstream of the base layer, and the second storage space is used to store the bitstream of the first enhancement layer. After obtaining the bitstream of the base layer of the first sub-image, the encoder can write the bitstream of the base layer of the first sub-image into the first storage space; and after obtaining the bitstream of the first enhancement layer of the first sub-image, it can write the bitstream of the first enhancement layer of the first sub-image into the second storage space.

[0039] Similarly, after obtaining the bitstreams of each image layer of other sub-images in the first frame image, the encoder can write the bitstreams of each image layer of other sub-images into their respective storage spaces.

[0040] Optionally, this application does not limit the storage location of the bitstreams of different sub-images in the same storage space of the first memory. In one possible implementation, the encoder can write the bitstreams of adjacent sub-images into adjacent address segments.

[0041] Furthermore, whenever the number of sub-images corresponding to the bitstream of the base layer stored in the first storage space reaches a preset first number, the encoder reads the bitstream of the base layer of the first number of sub-images and sends it to the decoder; similarly, whenever the number of sub-images corresponding to the bitstream of the first enhancement layer stored in the second storage space reaches the first number, the encoder reads the bitstream of the first enhancement layer of the first number of sub-images and sends it to the decoder.

[0042] In a Wi-Fi network, the encoding end needs to preempt the channel for data transmission before sending data to the decoding end each time. However, preemption of the channel can fail, or the success rate of preemption is relatively limited. Therefore, in this application, the bitstream of the same image layer of each sub-image is written into the off-chip memory in sequence. After a certain amount of data has been accumulated, it is sent to the decoding end. This reduces the number of times the channel is preempted, thereby improving the efficiency of data transmission.

[0043] Optionally, the encoder may also write the reconstructed images of each image layer of the first sub-image into the off-chip memory to generate an inter-frame reference image for the second frame image; or, the reconstructed images of each image layer of the first sub-image may be processed by loop filtering and then written into the off-chip memory to improve image quality. The second frame image is located after the first frame image in the time-frequency frame sequence.

[0044] Secondly, this application also provides an image decoding method, which can be used in an image decoding device. The image decoding device may include a system-on-a-chip (SOC) and an external memory. The SOC may include a decoder, an on-chip memory, and a communication interface. The decoder, the on-chip memory, and the communication interface are coupled to each other. The external memory is located outside the SOC and is coupled to the communication interface. The method may include: the decoder reading the bitstream of multiple image layers of a first sub-image of a first frame image from a first memory, and writing the bitstream of the multiple image layers of the first sub-image into a second memory, wherein the first memory is an off-chip memory and the second memory is an on-chip memory, the first frame image includes multiple sub-images, the multiple sub-images include the first sub-image, the multiple image layers include a base layer and a first enhancement layer, and the image quality of the first enhancement layer is higher than that of the base layer; the decoder reading the bitstream of the base layer of the first sub-image from the second memory; the decoder decoding the bitstream of the base layer of the first sub-image to obtain a first decoded image, the first decoded image being the decoded image of the base layer of the first sub-image; the decoder writing the first decoded image into the second memory; the decoder reading the bitstream of the first enhancement layer of the first sub-image and the first decoded image from the second memory; and the decoder decoding the bitstream of the first enhancement layer of the first sub-image based on the first decoded image to obtain a second decoded image, the second decoded image being the decoded image of the first enhancement layer of the first sub-image.

[0045] Optionally, the first memory can be DRAM, and the second memory can be SRAM.

[0046] Optionally, the first frame image mentioned above can be any frame in the video frame sequence, and this application embodiment does not limit this.

[0047] Optionally, the first sub-image can be any one of the plurality of sub-images, and this embodiment of the application does not limit this.

[0048] Optionally, the aforementioned sub-image can be, but is not limited to, a block, MB, or LCU within a frame image, or other block-level images; this application embodiment does not limit this. Here, a block-level image refers to an image that does not fully conform to the standard size specified by a block, macroblock, or maximum coding block, but whose size difference from the standard size of a block, macroblock, or maximum coding block is less than a preset size threshold.

[0049] In existing solutions, for any two adjacent (i.e., adjacent image quality) image layers of the image to be decoded, the decoder writes the decoded image of the lower (i.e. lower image quality) image layer to the off-chip memory. Correspondingly, the decoder needs to read the decoded image of the lower image layer from the off-chip memory and store it in the on-chip memory, and then read the decoded image of the lower image layer from the on-chip memory to decode the bitstream of the higher (i.e. higher image quality) image layer.

[0050] In other words, the decoder in the existing solution needs to frequently access the off-chip memory to obtain the decoded images of each image layer required for decoding. As the number of image layers increases, the number of reads and writes to the off-chip memory and the amount of data read and written will increase significantly. In addition, since the off-chip memory is usually characterized by slow read and write speed and high power consumption, the cost and power consumption of the decoder accessing the off-chip memory and the on-chip memory in the existing solution are relatively large.

[0051] In this application, for any two adjacent image layers of a sub-image to be decoded, the decoder writes the decoded image of the lower (i.e., lower quality) image layer into the on-chip memory. Correspondingly, the decoder can read only the decoded image of the lower image layer from the on-chip memory to decode the bitstream of the higher (i.e., higher quality) image layer. Therefore, the image decoding method provided in this application reduces the number of read / write operations and the amount of data read / written by the decoder to the external and on-chip memory. Furthermore, since the on-chip memory typically has fast read / write speeds and low power consumption, the image decoding method provided in this application reduces the cost and power consumption of the decoder accessing the external and on-chip memory.

[0052] In one possible implementation, the decoder can directly write the first decoded image into the on-chip memory. That is, the first decoded image is an image that has not undergone loop filtering.

[0053] In existing solutions, the decoded images of each image layer require loop filtering before being written to off-chip memory to improve image quality. However, while loop filtering improves image quality, it also increases the amount of data stored, requiring subsequent retrieval of the decoded images from this off-chip memory. This increases the decoder's bandwidth requirements for accessing off-chip memory, leading to longer image decoding times and reduced data transmission efficiency.

[0054] In this application, the decoded images of each image layer (such as the first decoded image) are directly stored in the on-chip memory without using loop filtering. This allows the first decoded image to be stored in the on-chip memory with a smaller data volume, and the decoded images of each image can be directly read from the on-chip memory for subsequent use. This reduces the bandwidth requirements of the decoder for accessing external memory, thereby reducing image decoding time and improving data transmission efficiency.

[0055] Optionally, after obtaining the second decoded image, the method further includes: the decoder writing the second decoded image into the second memory, and the second decoded image overwriting the first decoded image.

[0056] The image decoding method provided in this application stores the reconstructed images of each image layer in the second memory in a multiplexed manner. This allows more data to be stored in a fixed storage space, thereby improving the utilization rate of the on-chip memory and reducing the cost of the on-chip memory.

[0057] Optionally, the plurality of image layers further includes a second enhancement layer, the image quality of which is higher than that of the first enhancement layer. The method further includes: the decoder reading the bitstream of the second enhancement layer of the first sub-image and the second decoded image from the second memory; the decoder decoding the bitstream of the second enhancement layer of the first sub-image based on the second decoded image to obtain a third decoded image, the third decoded image being the decoded image of the second enhancement layer of the first sub-image; and the decoder writing the third decoded image into the second memory, wherein the third decoded image overwrites the second decoded image.

[0058] Optionally, the method further includes: after obtaining the second decoded image, the decoder writes the decoded image of the target image layer of the first sub-image into the first memory, wherein the target image layer is the image layer with the highest image quality among the plurality of image layers.

[0059] Furthermore, whenever the number of sub-images corresponding to the decoded images stored in the first memory reaches a preset first number, the decoder reads the decoded image of the first enhancement layer of the first number of sub-images and sends it to the display device.

[0060] In a Wi-Fi network, the decoding end needs to preempt the channel used for data transmission before sending data to the display device each time. However, preemption of the channel can fail, or the success rate of preemption is relatively limited. Therefore, in this application, the decoded image of the highest (i.e., the image with the highest image quality) image layer of each sub-image is written into the off-chip memory in sequence. After accumulating a certain amount of data, it is sent to the display device. This reduces the number of times the channel is preempted, thereby improving the efficiency of data transmission.

[0061] Optionally, the decoder can decode the bitstream of the multiple sub-images in a serial or parallel manner, and this application does not limit this.

[0062] In one possible implementation, the plurality of sub-images further includes a second sub-image, which is decoded adjacently to the first sub-image in the decoding order. Taking a serial approach as an example, the decoder can decode the bitstreams of the multiple image layers of the second sub-image after decoding the bitstreams of the multiple image layers of the first sub-image. In this way, all sub-images can be encoded layer by layer sequentially by a single encoder, which can reduce the cost of the encoder.

[0063] In one possible implementation, the plurality of sub-images further includes a second sub-image, which is decoded adjacently to the first sub-image in the order of decoding. For example, in a parallel manner, the decoder can decode the bitstream of the base layer of the second sub-image while simultaneously decoding the bitstream of the first enhancement layer of the first sub-image. This requires multiple encoders (such as those that may include multiple sub-encoders) to each complete the layered encoding of one sub-image, enabling parallel layered encoding of the plurality of sub-images, thus reducing encoding time.

[0064] Optionally, the decoder can also write the decoded images of each image layer of the first sub-image into the off-chip memory to generate an inter-frame reference image for the second frame; or, it can apply loop filtering to the decoded images of each image layer of the first sub-image before writing them into the off-chip memory to improve image quality. The second frame image is located after the first frame image in the time-frequency frame sequence.

[0065] It should be noted that since the decoding process is the reverse of the encoding process, the contents not described in detail in the second aspect and the beneficial effects can be referred to the relevant introduction in the first aspect, and will not be repeated here.

[0066] Thirdly, this application also provides an image encoding apparatus, which may include: a system-on-a-chip (SOC) and an off-chip memory. The SOC includes an encoder, an on-chip memory, and a communication interface. The encoder, the on-chip memory, and the communication interface are coupled to each other. The off-chip memory is located outside the SOC and is coupled to the communication interface. The encoder is used to implement the steps described in the first aspect or any possible implementation thereof.

[0067] Fourthly, this application also provides an image decoding apparatus, which may include: a system-on-chip (SOC) and an off-chip memory. The SOC includes a decoder, on-chip memory, and a communication interface. The decoder, on-chip memory, and communication interface are coupled to each other. The off-chip memory is located outside the SOC and is coupled to the communication interface. The decoder is used to implement the steps described in the second aspect above or any possible implementation thereof.

[0068] Fifthly, this application also provides an encoder that can be used in an image encoding apparatus. The encoder may include a read / write control module and an encoding module. The read / write control module is used to read a first sub-image of a first frame image from a first memory and store the first sub-image in a second memory. The first memory is an off-chip memory, and the second memory is an on-chip memory. The first frame image includes multiple sub-images, and the multiple sub-images include the first sub-image. The first sub-image is read from the second memory. The encoding module is used to encode the basic layer of the first sub-image based on the first sub-image to obtain the first sub-image. The image includes a base layer bitstream and a first reconstructed image, wherein the first reconstructed image is a reconstructed image of the base layer of the first sub-image; the read / write control module is further configured to write the first reconstructed image into the second memory; read the first reconstructed image and the first sub-image from the second memory; the encoding module is further configured to encode a first enhancement layer of the first sub-image based on the first reconstructed image and the first sub-image to obtain a first enhancement layer bitstream and a second reconstructed image of the first sub-image, wherein the second reconstructed image is a reconstructed image of the first enhancement layer of the first sub-image, and the image quality of the first enhancement layer is higher than that of the base layer.

[0069] In one possible implementation, the encoding module is specifically used to write the second reconstructed image directly into the second memory.

[0070] In one possible implementation, the read / write control module is further configured to write the second reconstructed image into the second memory after obtaining the second reconstructed image, and the second reconstructed image overwrites the first reconstructed image.

[0071] In one possible implementation, the read / write control module is further configured to read the second reconstructed image and the first sub-image from the second memory; the encoding module is further configured to encode the second enhancement layer of the first sub-image based on the second reconstructed image and the first sub-image to obtain the bitstream of the second enhancement layer of the first sub-image and a third reconstructed image, wherein the third reconstructed image is a reconstructed image of the second enhancement layer of the first sub-image, and the image quality of the second enhancement layer is higher than that of the first enhancement layer; the read / write control module is further configured to write the third reconstructed image to the second memory, and the third reconstructed image covers the second reconstructed image.

[0072] In one possible implementation, the first memory includes a first storage space and a second storage space. The first storage space is used to store the bitstream of the base layer, and the second storage space is used to store the bitstream of the first enhancement layer. The read / write control module is also used to write the bitstream of the base layer of the first sub-image into the first storage space and to write the bitstream of the first enhancement layer of the first sub-image into the second storage space.

[0073] In one possible implementation, the device may further include a transmitting module; the read / write control module is further configured to read the bitstream of the basic layer of the first number of sub-images whenever the number of sub-images corresponding to the bitstream of the basic layer stored in the first storage space reaches a preset first number; the transmitting module is configured to transmit the bitstream of the basic layer of the first number of sub-images to the decoding end; the read / write control module is further configured to read the bitstream of the first enhancement layer of the first number of sub-images whenever the number of sub-images corresponding to the bitstream of the first enhancement layer stored in the second storage space reaches the first number; the transmitting module is further configured to transmit the bitstream of the first enhancement layer of the first number of sub-images to the decoding end.

[0074] In one possible implementation, the plurality of sub-images further includes a second sub-image, and the second sub-image is adjacent to the encoding order of the first sub-image. The encoding module is further configured to encode the plurality of image layers of the second sub-image after completing the encoding of the plurality of image layers of the first sub-image, the plurality of image layers including the base layer and the first enhancement layer; or, the base layer of the second sub-image is encoded while the first enhancement layer of the first sub-image is encoded.

[0075] In one possible implementation, the sub-image is a block, a macroblock, or a maximum coded block.

[0076] In one possible implementation, the second memory is a static random access memory (SRAM), and the first memory is a dynamic random access memory (DRAM).

[0077] Sixthly, this application also provides a decoder that can be used in an image decoding device. The decoder may include: a read / write control module and a decoding module; the read / write control module is configured to read the bitstream of multiple image layers of a first sub-image of a first frame image from a first memory, and write the bitstream of the multiple image layers of the first sub-image into a second memory, wherein the first memory is an off-chip memory and the second memory is an on-chip memory, the first frame image includes multiple sub-images, the multiple sub-images include the first sub-image, the multiple image layers include a base layer and a first enhancement layer, and the image quality of the first enhancement layer is higher than that of the base layer; the decoding module is configured to decode the bitstream of the base layer of the first sub-image to obtain a first decoded image, the first decoded image being the decoded image of the base layer of the first sub-image; the read / write control module is further configured to write the first decoded image into the second memory; read the bitstream of the first enhancement layer of the first sub-image and the first decoded image from the second memory; the decoding module is further configured to decode the bitstream of the first enhancement layer of the first sub-image based on the first decoded image to obtain a second decoded image, the second decoded image being the decoded image of the first enhancement layer of the first sub-image.

[0078] In one possible implementation, the read / write control module is specifically used to directly write the first decoded image into the second memory.

[0079] In one possible implementation, the read / write control module is further configured to write the second decoded image into the second memory after obtaining the second decoded image, and the second decoded image overwrites the first decoded image.

[0080] In one possible implementation, the plurality of image layers further includes a second enhancement layer, the image quality of which is higher than that of the first enhancement layer. The read / write control module is further configured to read the bitstream of the second enhancement layer of the first sub-image and the second decoded image from the second memory. The decoding module is further configured to decode the bitstream of the second enhancement layer of the first sub-image based on the second decoded image to obtain a third decoded image, which is the decoded image of the second enhancement layer of the first sub-image. The read / write control module is further configured to write the third decoded image into the second memory, and the third decoded image overwrites the second decoded image.

[0081] In one possible implementation, the read / write control module is further configured to write the decoded image of the target image layer of the first sub-image into the first memory, wherein the target image layer is the image layer with the highest image quality among the plurality of image layers.

[0082] In one possible implementation, the device may further include a sending module, which is further configured to read the decoded image of the first enhancement layer of the first number of sub-images whenever the number of sub-images corresponding to the decoded images stored in the first memory reaches a preset first number; the sending module is configured to send the decoded image of the first enhancement layer of the first number of sub-images to a display device.

[0083] In one possible implementation, the plurality of sub-images further includes a second sub-image, which is adjacent to the first sub-image in the decoding order. The decoding module is further configured to decode the bitstream of the plurality of image layers of the second sub-image after decoding the bitstream of the plurality of image layers of the first sub-image; or, the decoding module is further configured to decode the bitstream of the base layer of the second sub-image while decoding the bitstream of the first enhancement layer of the first sub-image.

[0084] In one possible implementation, the sub-image is a block, a macroblock, or a maximum coded block.

[0085] In one possible implementation, the second memory is a static random access memory (SRAM), and the first memory is a dynamic random access memory (DRAM).

[0086] In a seventh aspect, this application also provides a computer storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect above or any possible implementation thereof, and / or the method described in the second aspect above or any possible implementation thereof.

[0087] Eighthly, this application also provides a computer program product that, when run on a processor, implements the method described in the first aspect above or any possible implementation thereof, and / or the method described in the second aspect above or any possible implementation thereof.

[0088] The image encoding device, image decoding device, encoder, decoder, computer-readable storage medium, or computer program product provided in the embodiments of this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here. Attached Figure Description

[0089] Figure 1 A schematic block diagram of a video encoding / decoding system 100 according to an embodiment of this application is provided;

[0090] Figure 2 A schematic block diagram of an image encoding apparatus 200 according to an embodiment of this application is provided;

[0091] Figure 3 A schematic block diagram of an image decoding apparatus 300 according to an embodiment of this application is provided;

[0092] Figure 4 A schematic flowchart of an image encoding method 400 according to an embodiment of this application is provided;

[0093] Figure 5 A schematic diagram of a frame image and multiple image layers of the frame image according to an embodiment of this application is provided;

[0094] Figure 6 A flowchart illustrating the multi-layer encoding process of a frame image according to an embodiment of this application is provided;

[0095] Figure 7 Another schematic diagram of the multi-layer encoding process for a frame of image according to an embodiment of this application is provided;

[0096] Figure 8 A schematic diagram of the storage space of the dynamic random access memory according to an embodiment of this application is provided;

[0097] Figure 9 An exemplary schematic diagram of the encoding process of the encoding end in an embodiment of this application is provided;

[0098] Figure 10 A schematic flowchart of an image decoding method 500 according to an embodiment of this application is provided;

[0099] Figure 11 A schematic block diagram of an encoder 600 according to an embodiment of this application is provided;

[0100] Figure 12 A schematic block diagram of a decoder 700 according to an embodiment of this application is provided. Detailed Implementation

[0101] The technical solution of this application will now be described with reference to the accompanying drawings.

[0102] The terms "first," "second," etc., used in the specification, embodiments, claims, and drawings of this application are for distinguishing purposes only and should not be construed as indicating or implying relative importance or order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0103] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0104] The technical solutions involved in the embodiments of this application may be applied not only to existing video coding standards (such as H.264 / Advanced Video Coding (AVC), H.265 / High Efficiency Video Coding (HEVC), etc.), but also to future video coding standards (such as H.266 / Versatile Video Coding (VVC)). The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. A brief introduction to some concepts that may be involved in the embodiments of this application follows.

[0105] In the field of video coding, the terms "picture," "frame," or "image" can be used synonymously. Video coding is performed on the source side and typically involves processing (e.g., by compression) the raw video picture to reduce the amount of data required to represent it, thereby enabling more efficient storage and / or transmission. Video decoding is performed on the destination side and typically involves inverse processing relative to the encoder to reconstruct the video picture. The term "coding" of video pictures in this embodiment should be understood as referring to the "coding" or "decoding" of a video sequence. The combination of encoding and decoding is also known as encoding / decoding.

[0106] The system architecture used in the embodiments of this application is described below.

[0107] Please refer to Figure 1 , Figure 1 A schematic block diagram of a video encoding / decoding system 100 for implementing embodiments of this application is shown. Figure 1As shown, the video encoding / decoding system 100 may include a source device 110 and a destination device 120. The source device 110 generates encoded video data; therefore, the source device 110 may be referred to as a video encoding device or encoding end. The destination device 120 can decode the encoded video data generated by the source device 110; therefore, the destination device 120 may be referred to as a video decoding device or decoding end.

[0108] In one possible implementation, both the source device 110 and the destination device 120 may include at least one processor and at least one memory coupled to the at least one processor.

[0109] Optionally, the processor may include, but is not limited to, a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor can implement or execute the methods and steps disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor.

[0110] Optionally, the memory may include, but is not limited to, random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or any other medium that can be used to store desired program code in the form of computer-accessible instructions or data structures.

[0111] The source device 110 and the destination device 120 can communicate via link 118, and the destination device 120 can receive encoded video data from the source device 110 via link 118.

[0112] Optionally, link 118 may include one or more media or devices capable of moving encoded video data from source device 110 to destination device 120.

[0113] In one possible implementation, link 118 may include one or more communication media that enable source device 110 to transmit encoded video data directly to destination device 120 in real time. In this example, source device 110 may modulate the encoded video data according to a communication standard (e.g., a wireless communication protocol) and transmit the modulated video data to destination device 120.

[0114] Optionally, the one or more communication media may include wireless and / or wired communication media, which is not limited in this application.

[0115] For example, the one or more communication media may include radio frequency (RF) spectrum or one or more physical transmission lines.

[0116] For example, the one or more communication media may form part of a packet-based network, which may be, for example, a local area network, a wide area network, or a global network (e.g., the Internet).

[0117] For example, the one or more communication media may include routers, switches, base stations, or other devices that facilitate communication from source device 110 to destination device 120.

[0118] The source device 110 may include an image encoding device 111. Optionally, the source device 110 may also include an image source 112, an image preprocessor 114, and a communication interface 117.

[0119] Image source 112 may include or be any type of image capture device used for, for example, capturing real-world images, and / or any type of image or commentary generation device (for screen content encoding, some text on the screen is also considered part of the image or picture to be encoded).

[0120] In one possible implementation, image source 112 may include a computer graphics processor for generating computer-animated images, or any type of device for acquiring and / or providing real-world images, computer-animated images (e.g., screen content, virtual reality (VR) images), and / or any combination thereof (e.g., augmented reality (AR) images).

[0121] In another possible implementation, image source 112 may include a camera for capturing images, or a memory for storing images, or any type of (internal or external) interface for storing previously captured or generated images and / or acquiring or receiving images.

[0122] For example, when the image source 112 is a camera, the image source 112 can be a local camera or an integrated camera integrated into the source device. When the image source 112 is a memory, the image source 112 can be a local memory or an integrated memory integrated into the source device. When the image source 112 includes an interface, the interface can be an external interface for receiving images from an external video source, such as an external image capture device, like a camera, external memory, or an external image generation device, such as an external computer graphics processor, computer, or server. The interface can be any type of interface according to any proprietary or standardized interface protocol, such as a wired or wireless interface, or an optical interface.

[0123] Image preprocessor 114 is used to receive raw image data 113 and perform preprocessing on the raw image data 113 to obtain a preprocessed image 115 or preprocessed image data 115.

[0124] Optionally, the preprocessing performed by the image preprocessor 114 may include retouching, color format conversion, color correction, or noise reduction.

[0125] It should be noted that performing preprocessing on image data 113 is not a mandatory process in this application, and this application does not impose specific restrictions on it.

[0126] Image encoding device 111 (or video encoding device 111) is used to receive preprocessed image data 115 and process the preprocessed image data 115 using a relevant prediction mode (such as the prediction mode in the various embodiments herein) to provide encoded image data 116. In some embodiments, image encoding device 111 can be used to perform the various embodiments described below to implement the image encoding method described in this application on the encoding side.

[0127] Communication interface 117 can be used to receive encoded image data 116 and transmit the encoded image data 116 to destination device 120 or any other device (such as a memory) via link 118 for storage or direct reconstruction. The other device can be any device used for decoding or storage. Communication interface 117 can also be used to encapsulate the encoded image data 116 into a suitable format, such as data packets, for transmission over link 118.

[0128] The destination device 120 includes an image decoding device 121. Optionally, the destination device 120 may also include a communication interface 122, an image post-processor 124, and a display device 125. These are described below:

[0129] Communication interface 122 can be used to receive encoded image data 116 from source device 110 or any other source, such as a storage device, for example, an encoded image data storage device. Communication interface 122 can be used to transmit or receive encoded image data 116 via link 118 between source device 110 and destination device 120 or via any type of network, such as a direct wired or wireless connection, or any type of network, such as a wired or wireless network or any combination thereof, or any type of private and public network, or any combination thereof. Communication interface 122 can be used, for example, to decapsulate data packets transmitted by communication interface 117 to obtain encoded image data 116.

[0130] Both communication interface 122 and communication interface 117 can be configured as unidirectional or bidirectional communication interfaces, and can be used, for example, to send and receive messages to establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission, such as encoded image data transmission.

[0131] Image decoding device 121 (or video decoding device 121) is used to receive encoded image data 116 and provide decoded image data 123 or decoded image 123. In some embodiments, image decoding device 121 can be used to perform various embodiments described below to implement the image decoding method described in this application on the decoding side.

[0132] Image post-processor 124 is used to perform post-processing on decoded image data 123 (also known as reconstructed image data) to obtain post-processed image data 125.

[0133] Optionally, the post-processing performed by the image post-processor 124 may include color format conversion, color correction, retouching or resampling, or any other processing, and may also be used to transmit the post-processed image data 125 to the display device 126.

[0134] It should be noted that performing post-processing on the decoded image data 123 (also known as reconstructed image data) is not a mandatory process in this application, and this application does not impose specific limitations on it.

[0135] Display device 126 is used to receive post-processed image data 125 to display the image to, for example, a user or viewer.

[0136] In one possible implementation, the display device 126 can be or may include any type of display for presenting the reconstructed image.

[0137] For example, display device 126 can be an integrated or external display or monitor.

[0138] Alternatively, the display may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0139] Optionally, the source device 110 and the destination device 120 can be various devices that provide video / data connectivity to users, and this application does not limit them.

[0140] For example, source device 110 and destination device 120 may include any of the following various devices, including any category of handheld or stationary devices, such as laptops or notebook computers, mobile phones, smartphones, tablets or tablet computers, wearable devices, cameras, desktop computers, set-top boxes, televisions, cameras, in-vehicle devices, display devices, digital media players, video game consoles, video streaming devices (such as content service servers or content distribution servers), broadcast receiver devices, broadcast transmitter devices, etc., and may or may not use any category of operating system.

[0141] In one possible implementation, the video encoding / decoding system 100 can be applied to short-range wireless projection scenarios. Short-range wireless projection refers to a technology that encodes and compresses video data generated by a mobile terminal with strong processing capabilities, and then wirelessly transmits it to a device with weaker processing capabilities but better display quality for display.

[0142] Example: Source device 110 is a mobile phone, and destination device 120 is VR glasses. The mobile phone can project the rendered game screen onto the display interface of the VR glasses.

[0143] For example: the source device 110 can be a laptop computer, and the destination device 120 can be a television. The laptop computer can project the television screen onto the television's display interface.

[0144] It should be noted that, Figure 1In this application, source device 110 and destination device 120 are depicted as separate devices, but this application is not limited thereto. Optionally, source device 110 and destination device 120 may be (or integrated into) the same device, that is, the device may simultaneously include the functionality corresponding to source device 110 and the functionality corresponding to destination device 120. In such embodiments, the same hardware and / or software, or separate hardware and / or software, or any combination thereof, may be used to implement the functionality corresponding to source device 110 and the functionality corresponding to destination device 120.

[0145] Example: Source device 110 and destination device 120 are integrated into a mobile phone, which can project the rendered game screen onto the phone's display interface.

[0146] It should be noted that, Figure 1 The image encoding device 111, image source 112, image preprocessor 114, and communication interface 117 are illustrated as independent components, but this application is not limited thereto. Optionally, the image source 112, image preprocessor 114, and / or communication interface 117 may be integrated into the image encoding device 111, or the image encoding device 111 may also have the functionality of the image source 112, image preprocessor 114, and / or communication interface 117.

[0147] Similarly, Figure 1 The image decoding device 121, communication interface 122, image post-processor 124, and display device 126 are illustrated as independent components, but this application is not limited thereto. Optionally, the communication interface 122 and / or the image post-processor 124 may be integrated into the image decoding device 121, or the image decoding device 121 may also have the functionality of the communication interface 122 and / or the image post-processor 124.

[0148] In some cases, Figure 1 The video encoding / decoding system 100 shown is merely an example, and the techniques of this application can be applied to video encoding setups (e.g., video encoding or video decoding) that do not necessarily involve any data communication between the source device 110 and the destination device 120. In other instances, data may be retrieved from local storage, streamed over a network, etc. A video encoding device may encode data and store it in storage, and / or a video decoding device may retrieve data from storage and decode it. In some instances, encoding and decoding are performed by devices that do not communicate with each other but only encode data into storage and / or retrieve data from storage and decode the data.

[0149] Optionally, the video codec system 100 described above may be applicable to, but is not limited to, the H.264 Scalable Video Coding (SVC) protocol, the H.265 Scalable High-Efficiency Video Coding (SHVC) protocol, or next-generation video standard protocols (such as H.266).

[0150] Please refer to Figure 2 , Figure 2 A schematic block diagram of an image encoding device 200 provided in an embodiment of this application is shown. The image encoding device 200 can be the image encoding device 111 in the video encoding and decoding system 100 described above. Figure 2 As shown, the image encoding device 200 may include a system on chip (SOC) 210 and an off-chip memory 220. The SOC 210 may include an encoder 211, an on-chip memory 212 and a communication interface 213. The encoder 211, the on-chip memory 212 and the communication interface 213 are coupled to each other, and the communication interface 213 is coupled to the off-chip memory 220.

[0151] The encoder 211 is used to run computer programs or instructions to enable the image encoding apparatus 200 to implement the image encoding method described in the embodiments of this application.

[0152] Optionally, the encoder 211 includes one or more processing units, including but not limited to a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, a microcontroller, or any conventional processor.

[0153] It should be noted that the on-chip memory 212 is located inside the SOC 210 and features fast read / write speed, small capacity, and low power consumption. The off-chip memory 220 is located outside or on the outer side of the SOC 210 (i.e., the off-chip memory 220 is not part of the SOC 210) and features slow read / write speed, large capacity, and high power consumption.

[0154] In one possible implementation, the on-chip memory can be Static Random Access Memory (SRAM), and the off-chip memory can be Dynamic Random Access Memory (DRAM). For example, SRAM can be a cache memory, and DRAM can be main memory.

[0155] External memory 220 is used to store the image to be encoded (e.g. Figure 1 Image data 115 in the image is also used to store the bitstream of multiple image layers obtained after the image to be encoded is layered by encoder 211 (e.g., Figure 1 Image data 116 in the image.

[0156] The encoder 211 can read the image to be encoded from the external memory 220 through the communication interface 213 and store the image to be encoded into the on-chip memory 212. It can also write the bitstream of the encoded multiple image layers into the external memory 220 through the communication interface 213.

[0157] The on-chip memory 212 is used to store the image to be encoded, and also to store the reconstructed images of the multiple image layers.

[0158] The encoder 211 can also read the image to be encoded and the reconstructed images of each image layer of the image to be encoded from the on-chip memory 212.

[0159] It should be noted that, Figure 2 Only a simplified design of the image encoding device 200 is shown. In practical applications, the image encoding device 200 may also include other necessary components, including but not limited to any number of communication interfaces, processors, controllers, memory, etc., and all image encoding devices that can implement this application are within the protection scope of this application.

[0160] Please refer to Figure 3 , Figure 3 A schematic block diagram of an image decoding device 300 provided in an embodiment of this application is shown. The image decoding device 300 can be the image decoding device 121 in the video encoding and decoding system 100 described above. Figure 3 As shown, the image decoding device 300 may include a SOC 310 and an off-chip memory 320. The SOC 310 may include a decoder 311, an on-chip memory 312 and a communication interface 313. The decoder 311, the on-chip memory 312 and the communication interface 313 are coupled to each other, and the communication interface 313 is coupled to the off-chip memory 320.

[0161] The decoder 311 is used to run computer programs or instructions to enable the image decoding device 300 to implement the image decoding method described in the embodiments of this application.

[0162] Optionally, the decoder 311 includes one or more processing units, which include, but are not limited to, CPUs, general-purpose processors, DSPs, ASICs, FPGAs, discrete gate or transistor logic devices or discrete hardware components. A general-purpose processor may be a microprocessor, a microcontroller, or any conventional processor.

[0163] It should be noted that the on-chip memory 312 is located inside the SOC 310 and features fast read / write speed, small capacity, and low power consumption. The off-chip memory 320 is located outside or on the outer side of the SOC 310 (i.e., the off-chip memory 320 is not part of the SOC 310) and features slow read / write speed, large capacity, and high power consumption.

[0164] In one possible implementation, the on-chip memory can be SRAM, and the off-chip memory can be DRAM. For example, the SRAM can be a cache memory, and the DRAM can be main memory.

[0165] External memory 320 is used to store the bitstream of multiple image layers (e.g. Figure 1 The image data 116 in the image data is also used to store the decoded images of the multiple image layers obtained after the bitstream of the multiple image layers is decoded by the decoder 311 (e.g., the decoded images of the multiple image layers). Figure 1 Image data 123 in the image.

[0166] The decoder 311 can read the bitstream of the multiple image layers from the external memory 320 through the communication interface 313 and store the bitstream of the multiple image layers into the on-chip memory 312. It can also write the decoded image of the multiple image layers obtained by decoding into the external memory 320 through the communication interface 313.

[0167] The on-chip memory 312 is used to store the bitstream of the multiple image layers and also to store the decoded images of the multiple image layers.

[0168] The decoder 311 can also read the bitstream of the multiple image layers and the decoded images of the multiple image layers from the on-chip memory 312.

[0169] It should be noted that, Figure 3 Only a simplified design of the image decoding device 300 is shown. In practical applications, the image decoding device 300 may also include other necessary components, including but not limited to any number of communication interfaces, processors, controllers, memory, etc., and all image decoding devices that can implement this application are within the protection scope of this application.

[0170] The image encoding method and image decoding method provided in the embodiments of this application are described in detail below.

[0171] Figure 4 A schematic flowchart of an image encoding method 400 provided in an embodiment of this application is shown. Method 400 can be used in the image encoding apparatus 200 described above. Method 400 is described as a series of steps or operations, but it should be understood that method 400 can be performed in various orders and / or occur simultaneously, and is not limited to... Figure 4 The execution order is shown. Figure 4 As shown, method 400 may include the following steps.

[0172] Step 401: The encoder reads the first sub-image of the first frame image from the first memory and stores the first sub-image in the second memory. The first memory is an off-chip memory and the second memory is an on-chip memory. The first frame image includes multiple sub-images, and the multiple sub-images include the first sub-image.

[0173] In other words, the encoder writes the first sub-image into the second memory.

[0174] It should be noted that the encoder and the second memory mentioned in step 401 can be located inside the SOC, while the first memory mentioned in step 401 can be located outside the SOC.

[0175] For example, the above SOC can be Figure 2 The encoder in the SOC 210 shown above can be... Figure 2 The encoder 211 shown above, the second memory can be Figure 2 The on-chip memory 212 shown above, the first memory can be... Figure 2 The off-chip memory 220 shown.

[0176] For example, the first memory can be DRAM and the second memory can be SRAM.

[0177] Optionally, the first frame image mentioned above can be any frame in the video frame sequence, and this application embodiment does not limit this.

[0178] Optionally, the first sub-image can be any one of the plurality of sub-images, and this embodiment of the application does not limit this.

[0179] Optionally, the aforementioned sub-image can be, but is not limited to, a block, macroblock (MB), or largest coding unit (LCU) in a frame image, or other block-level images. This application embodiment does not limit this. Here, a block-level image refers to an image that does not fully conform to the standard size specified by a block, macroblock, or largest coding unit, but whose size difference from the standard size of the block, macroblock, or largest coding unit is less than a preset size threshold.

[0180] It should be noted that each sub-image at the encoding end can be divided into a base layer and at least one enhancement layer. The base layer provides the most basic image quality, and the enhancement layers improve and enhance the image quality based on the base layer; that is, the image quality of the enhancement layers is higher than that of the base layer. The image quality of the at least one enhancement layer is arranged in ascending order. Here, "image quality" as described in this application may include pixel fidelity, and optionally, "image quality" may also include resolution and / or frame rate, meaning the base layer has the lowest image quality.

[0181] Example, Figure 5 A schematic diagram of a frame image and its multiple image layers is shown, such as... Figure 5 As shown, image 1 comprises four sub-images: B1, B2, B3, and B4. Image 1 is divided into three image layers: a base layer, enhancement layer 1, and enhancement layer 2. Taking B1 as an example, when the encoder performs layered encoding on B1, it first encodes the bitstream of the base layer (B1-0) and its reconstructed image. Then, using B1 and the reconstructed image of B1-0 as a reference, it encodes the bitstream of enhancement layer 1 (B1-1) and its reconstructed image. Finally, using B1 and the reconstructed image of B1-1 as a reference, it encodes the bitstream of enhancement layer 2 (B1-2) and its reconstructed image. Therefore, the image quality of B1-0 is lower than that of B1-1, and the image quality of B1-1 is lower than that of B1-2. Figure 5 The relationship between the image quality of each image layer in B2, B3, and B4 can be referred to in B1, and will not be repeated here. It should be noted that the layered encoding of sub-images described in this application can also be referred to as encoding multiple image layers of a sub-image.

[0182] Correspondingly, the more image layers the decoding end receives (or decodes), the higher the image quality. When propagating an SVC-encoded bitstream in a communication network, different modulation and coding schemes (MCS) can be used for different image layers. For example, using a low-rate MCS for the basic layer and low-level enhancement layers can enable user equipment with poor channel conditions to obtain basic video services, while using a high-rate MCS for high-level enhancement layers can enable user equipment with good channel conditions to obtain higher quality and higher resolution video services.

[0183] Step 402, the encoder reads the first sub-image from the on-chip memory.

[0184] Step 403: The encoder encodes the basic layer of the first sub-image based on the first sub-image to obtain the bitstream of the basic layer of the first sub-image and the first reconstructed image, wherein the first reconstructed image is the reconstructed image of the basic layer of the first sub-image.

[0185] Optionally, the encoding method of the basic layer of the first sub-image can be intra-frame coding or inter-frame coding, and this application does not limit it.

[0186] In one possible implementation, if the first frame image is the first frame image in the video sequence, then step 403 may include: encoding the basic layer of the first sub-image using intra-frame coding based on the first sub-image and the intra-frame reference image.

[0187] Optionally, this application does not limit the method of obtaining the aforementioned intra-frame reference image. For example, the intra-frame reference image can be obtained by referring to existing technologies.

[0188] In another possible implementation, if the first frame image is not the first frame image in the video sequence, step 403 may include: encoding the basic layer of the first sub-image using inter-frame coding based on the first sub-image and the inter-frame reference image.

[0189] Optionally, this application does not limit the method of obtaining the inter-frame reference image. For example, the inter-frame reference image can be obtained by referring to existing technologies.

[0190] In step 404, the encoder writes the first reconstructed image into the on-chip memory.

[0191] Since the encoder has already written the first sub-image to the on-chip memory in step 401, meaning the first sub-image is already stored in the on-chip memory, in step 404 the encoder only needs to write the first reconstructed image to the on-chip memory again. In this way, both the first sub-image and the first reconstructed image are stored in the on-chip memory, and can be read directly from the on-chip memory when used later.

[0192] It should be noted that, since the sub-image described in this application is a block-level image, the data volume of the block-level image is much smaller than the storage space of the on-chip memory. Therefore, the encoder can write the sub-image to be encoded and the reconstructed images of each image layer of the sub-image into the on-chip memory, and can directly read them from the on-chip memory when used later.

[0193] In one possible implementation, the encoder can directly write the first reconstructed image into the on-chip memory. That is, the first reconstructed image is an image that has not undergone loop filtering.

[0194] In existing solutions, the reconstructed images of each image layer require in-loop filtering before being written to off-chip memory to improve image quality. However, while in-loop filtering improves image quality, it also increases the amount of data stored, requiring subsequent retrieval of the reconstructed images from this off-chip memory. This increases the encoder's bandwidth requirements for accessing off-chip memory, leading to longer image encoding times and reduced data transmission efficiency.

[0195] In this application, the reconstructed images of each image layer (such as the first reconstructed image) are directly stored in the on-chip memory without using loop filtering. This allows the first reconstructed image to be stored in the on-chip memory with a smaller data volume, and the reconstructed images of each image can be directly read from the on-chip memory for subsequent use. This reduces the encoder's bandwidth requirements for accessing external memory, thereby reducing image encoding time and improving data transmission efficiency.

[0196] Optionally, this application does not limit the storage location of the first sub-image and the first reconstructed image in the on-chip memory.

[0197] For example, the on-chip memory may include storage space a and storage space b, which do not overlap, wherein the first sub-image is written to storage space a and the first reconstructed image is written to storage space b.

[0198] Step 405: The encoder reads the first reconstructed image and the first sub-image from the on-chip memory.

[0199] It should be noted that since the first sub-image and the first reconstructed image are stored in the on-chip memory after step 404, the encoder can directly read the first sub-image and the first reconstructed image from the on-chip memory in step 405.

[0200] In other words, the first sub-image is read from the off-chip memory only once when it is used for the first time, and can be read directly from the on-chip memory for subsequent uses.

[0201] Step 406: The encoder encodes the first enhancement layer of the first sub-image based on the first reconstructed image and the first sub-image to obtain the bitstream of the first enhancement layer of the first sub-image and the second reconstructed image. The second reconstructed image is the reconstructed image of the first enhancement layer of the first sub-image, and the image quality of the enhancement layer is greater than that of the base layer.

[0202] In one possible implementation, the encoder can encode the enhancement layer of the first sub-image using inter-layer coding based on the first reconstructed image and the first sub-image. That is, the first reconstructed image is a reference image for inter-layer coding (from the base layer to the enhancement layer).

[0203] Optionally, after obtaining the second reconstructed image through step 406, the method 400 further includes: the encoder writing the second reconstructed image into the second memory.

[0204] Optionally, this application does not limit the storage location of the second reconstructed image in the on-chip memory.

[0205] In one possible implementation, the on-chip memory may include storage space a and storage space b, which do not overlap. During the layered encoding process of the same sub-image, storage space a is used to store the first sub-image, and storage space b is used to store the reconstructed images of each image layer in a multiplexed manner. This allows more data to be stored in a fixed amount of storage space, thereby improving the utilization rate of the on-chip memory and reducing its cost.

[0206] For example, if the first sub-image is written to storage space a and the first reconstructed image is written to storage space b, the second reconstructed image can be written to storage space b and overwrite the first reconstructed image.

[0207] In another possible implementation, the on-chip memory may include multiple storage spaces (e.g., storage space a, storage space b, and storage space c), which do not overlap. During the hierarchical encoding of the same sub-image, different storage spaces are used to store different images. This reduces the complexity of data reading and writing, thereby improving data transmission efficiency.

[0208] For example, if the first sub-image is written to storage space a, the first reconstructed image is written to storage space b, and the second reconstructed image can be written to storage space c.

[0209] Optionally, steps 401 to 406 are described using only the encoding of the basic layer and the encoding of the first enhancement layer of the first sub-image as examples, but this application is not limited thereto. Optionally, the image encoding device can also use a similar method to encode other image layers (image layers with higher image quality), and this application embodiment does not limit this.

[0210] In one possible implementation, the encoder can read the second reconstructed image and the first sub-image from the second memory; based on the second reconstructed image and the first sub-image, it encodes the second enhancement layer of the first sub-image to obtain the bitstream of the second enhancement layer of the first sub-image and a third reconstructed image, wherein the third reconstructed image is the reconstructed image of the second enhancement layer of the first sub-image, and the image quality of the second enhancement layer is higher than that of the first enhancement layer; the third reconstructed image is written to the second memory, and the third reconstructed image covers the second reconstructed image.

[0211] In other words, the reconstructed images of each image layer reuse a fixed storage space in the on-chip memory. This allows more data to be stored in a fixed storage space, thereby improving the utilization rate of the on-chip memory and reducing its cost.

[0212] The above description allows for the encoding of multiple image layers (such as the base layer, the first enhancement layer, and the second enhancement layer) of the first sub-image. Furthermore, the encoder can use a similar method to encode multiple image layers of other sub-images within these multiple sub-images, which will not be elaborated upon here.

[0213] For example, such as Figure 5 As shown, the encoder can follow Figure 5 The sub-images in image 1 are read sequentially in the order indicated by the middle arrow, and the read sub-images are then encoded in layers.

[0214] In one possible implementation, if the plurality of sub-images further includes a second sub-image, and the encoding order of the second sub-image and the first sub-image is adjacent, the method 400 may further include: after completing the encoding of the plurality of image layers of the first sub-image, encoding the plurality of image layers of the second sub-image, the plurality of image layers including the base layer and the first enhancement layer.

[0215] In other words, the layered encoding of multiple sub-images is performed serially. This allows a single encoder to sequentially complete the layered encoding of all sub-images, reducing encoder costs.

[0216] In another possible implementation, if the plurality of sub-images also includes a second sub-image, and the encoding order of the second sub-image and the first sub-image is adjacent, the method 400 may further include: encoding the base layer of the second sub-image while encoding the first enhancement layer of the first sub-image.

[0217] In other words, the hierarchical encoding of multiple sub-images is performed in parallel. This requires multiple encoders (which may include multiple sub-encoders) to each complete the hierarchical encoding of one sub-image, so that the hierarchical encoding of multiple sub-images can be performed in parallel, which can reduce the encoding time.

[0218] Please refer to Figure 6 Taking image 1 as an example, which includes four sub-images, namely B1, B2, B3, and B4, and performing layered encoding of each sub-image across three image layers: a base layer, enhancement layer 1, and enhancement layer 2, with the four sub-images being encoded in the serial manner described above, Figure 6 A schematic diagram of the hierarchical coding process provided in this application is shown. For example... Figure 6 As shown, the hierarchical coding process mainly includes the following steps.

[0219] Step 1. The encoder reads B1 from DRAM and stores it in SRAM storage space a, such as... Figure 6 As shown in (11) of the table.

[0220] Step 2. The encoder reads B1 from storage space a, such as... Figure 6 As shown in (12) of the table.

[0221] Step 3. The encoder encodes the basic layer (i.e., B1-0) of B1 based on B1 to obtain the bitstream of B1-0 and the reconstructed image of B1-0.

[0222] Step 4. The encoder writes the reconstructed image of B1-0 into the SRAM storage space b, such as... Figure 6 As shown in (13) of the table.

[0223] Step 5. The encoder reads B1 from storage space a, such as... Figure 6 As shown in (14-1); and read the reconstructed image of B1-0 from the storage space b, as shown in (14-1). Figure 6 As shown in (14-2) in the document. Optionally, this application does not limit the execution order of (14-1) and (14-2).

[0224] Step 6. Based on the reconstructed images of B1 and B1-0, the encoder encodes the enhancement layer 1 (i.e. B1-1) of B1 to obtain the bitstream of B1-1 and the reconstructed image of B1-1.

[0225] Step 7. The encoder writes the reconstructed image of B1-1 into the storage space b and overwrites the reconstructed image of B1-0, as shown. Figure 6 As shown in (15) of the table.

[0226] Step 8. The encoder reads B1 from storage space a, such as... Figure 6 As shown in (16-1); and read the reconstructed image of B1-1 from the storage space b, as shown in (16-1). Figure 6 As shown in (16-2) in the document. Optionally, this application does not limit the execution order of (16-1) and (16-2).

[0227] Step 9. Based on the reconstructed images of B1 and B1-1, the encoder encodes the enhancement layer 2 (i.e., B1-2) of B1 to obtain the bitstream of B1-2 and the reconstructed image of B1-2.

[0228] Steps 1 through 9 above have described the process of three-layer encoding of B1. Furthermore, a similar method can be used to sequentially complete the three-layer encoding of B2, B3, and B4, thereby achieving three-layer encoding of B1, B2, B3, and B4 in a serial manner. For examples, please refer to steps 1 through 9 above. Figure 6 The three-layer encoding of B2 is completed by (21), (22), (23), (24-1), (24-2), (25), (26-1), and (26-2).

[0229] Please refer to Figure 7 Taking image 1 as an example, which includes four sub-images, namely B1, B2, B3, and B4, and performing layered encoding of each sub-image using three image layers: a base layer, enhancement layer 1, and enhancement layer 2, with the four sub-images being encoded in a parallel manner as described above. Figure 7 Another flowchart illustrating the hierarchical coding provided in this application is shown. For example... Figure 7 As shown, the hierarchical coding process mainly includes the following steps.

[0230] Step a. The basic layer encoder reads B1 from DRAM and stores it in storage space a of SRAM, as shown. Figure 6 As shown in (11) of the table.

[0231] Step b. The base layer encoder reads B1 from storage space a, as follows: Figure 6 As shown in (12) in the diagram. The basic layer encoder reads B2 from DRAM and stores it in the memory space a, as shown in the diagram. Figure 6 As shown in (21) of the table.

[0232] Step c. The base layer encoder encodes the base layer (i.e., B1-0) of B1 based on B1, obtaining the bitstream of B1-0 and the reconstructed image of B1-0. The base layer encoder reads B2 from storage space a, as follows. Figure 6 As shown in (22) in the middle.

[0233] Step d. The base layer encoder writes the reconstructed image of B1-0 into the SRAM storage space b, as shown. Figure 6 As shown in (13) in the figure. The basic layer encoder encodes the basic layer (i.e., B2-0) of B2 based on B2 to obtain the bitstream of B2-0 and the reconstructed image of B2-0.

[0234] Step e. The enhancement layer 1 encoder reads B1 from storage space a, as follows: Figure 6 As shown in (14-1); and read the reconstructed image of B1-0 from the storage space b, as shown in (14-1). Figure 6 As shown in (14-2). The basic layer encoder writes the reconstructed image of B2-0 into the storage space b of the SRAM, as shown. Figure 6 As shown in (23) in the middle.

[0235] Optionally, this application does not limit the execution order of (14-1) and (14-2).

[0236] Step f. The enhancement layer 1 encoder encodes enhancement layer 1 (i.e., B1-1) of B1 based on the reconstructed images of B1 and B1-0, obtaining the bitstream of B1-1 and the reconstructed image of B1-1. The enhancement layer 1 encoder reads B2 from storage space a, as follows. Figure 6 As shown in (24-1); and read the reconstructed image of B2-0 from the storage space b, as shown in (24-1). Figure 6 As shown in (24-2) in the figure.

[0237] Optionally, this application does not limit the execution order of (24-1) and (24-2).

[0238] Step g. The encoder of enhancement layer 1 writes the reconstructed image of B1-1 into the storage space b and overwrites the reconstructed image of B1-0, as shown. Figure 6 As shown in (15). The enhancement layer 1 encoder encodes the enhancement layer 1 (i.e., B2-1) of B2 based on the reconstructed images of B2 and B2-0, to obtain the bitstream of B2-1 and the reconstructed image of B2-1.

[0239] Step h. The enhancement layer 2 encoder reads B1 from the storage space a, such as Figure 6 As shown in (16-1); and read the reconstructed image of B1-1 from the storage space b, as shown in (16-1). Figure 6As shown in (16-2). The encoder of enhancement layer 1 writes the reconstructed image of B2-1 into the storage space b and overwrites the reconstructed image of B2-0, as shown in Figure 16-2. Figure 6 As shown in (25) of the table.

[0240] Optionally, this application does not limit the execution order of (16-1) and (16-2).

[0241] Step i. The enhancement layer 2 encoder encodes enhancement layer 2 (i.e., B1-2) of B1 based on the reconstructed images of B1 and B1-1, obtaining the bitstream of B1-2 and the reconstructed image of B1-2. The enhancement layer 2 encoder reads B2 from the storage space a, as follows. Figure 6 As shown in (26-1); and read the reconstructed image of B2-1 from the storage space b, as shown in (26-1). Figure 6 As shown in (26-2) in the figure.

[0242] Optionally, this application does not limit the execution order of (26-1) and (26-2).

[0243] Step j. The enhancement layer 2 encoder encodes the enhancement layer 2 (i.e., B2-2) of B2 based on the reconstructed images of B2 and B2-1, to obtain the bitstream of B2-2 and the reconstructed image of B2-2.

[0244] Steps a through j above described the process of performing three-layer encoding on B1 and B2 in parallel. Similarly, a similar method can be used to perform three-layer encoding on B1, B2, B3, and B4 in parallel. For example, B3 is read from DRAM starting in step c, and B4 is read from DRAM starting in step d.

[0245] In existing solutions, before encoding each image layer of the image to be encoded, the encoder needs to read the image to be encoded from the off-chip memory and store it in the on-chip memory, and then read the image to be encoded from the on-chip memory for encoding. Furthermore, for any two adjacent (i.e., adjacent image quality) image layers of the image to be encoded, the encoder writes the reconstructed image of the lower (i.e., lower image quality) image layer to the off-chip memory. Correspondingly, the encoder needs to read the reconstructed image of the lower image layer from the off-chip memory and store it in the on-chip memory, and then read the reconstructed image of the lower image layer from the on-chip memory to encode the higher (i.e., higher image quality) image layer.

[0246] In other words, the encoder in the existing solution needs to frequently access the off-chip memory to obtain the image data required for encoding and / or store the encoded image data. As the number of image layers increases, the number of reads and writes to the off-chip memory and the amount of data read and written will increase significantly. In addition, since the off-chip memory is usually characterized by slow read and write speed and high power consumption, the cost and power consumption of the encoder in the existing solution for accessing the off-chip memory and the on-chip memory are relatively large.

[0247] And through Figure 6 and Figure 7 As can be seen in this application, the encoder only needs to read the sub-image from the DRAM and store it in the SRAM once when it first obtains the sub-image to be encoded, such as... Figure 6 and Figure 7 As shown in (11) or as shown in Figure 6 and Figure 7 As shown in (21), during the subsequent encoding of multiple image layers of this sub-image, only this sub-image is read from this SRAM for encoding, as follows: Figure 6 and Figure 7 As shown in (12), (14-1) and (16-1) or as shown in Figure 6 and Figure 7 As shown in (22), (24-1), and (26-1) in the diagram. Furthermore, for any two adjacent image layers, the encoder writes the reconstructed image of the lower (i.e., lower quality) image layer into the SRAM, as shown in (22), (24-1), and (26-1). Figure 6 and Figure 7 As shown in (13) and (15) or as shown in Figure 6 and Figure 7 As shown in (23) and (25) in the figure; accordingly, the encoder can read only the reconstructed image of the lower image layer from the SRAM to encode the higher (i.e., higher image quality) image layer, as ... Figure 6 and Figure 7 As shown in (14-2) and (16-2) or as shown in Figure 6 and Figure 7 As shown in (24-2) and (26-2) in the figure.

[0248] As can be seen, the image encoding method provided in this application can reduce the number of read / write operations and the amount of data read / written to the external memory and the internal memory. In addition, since the internal memory usually has the characteristics of fast read / write speed and low power consumption, the image encoding method provided in this application can reduce the cost and power consumption of the encoder accessing the external memory and the internal memory.

[0249] Optionally, the first memory may include multiple storage spaces, with different storage spaces used to store bitstreams of different image layers. The method 400 further includes: the encoder writing the bitstreams of different image layers into the storage space corresponding to each image layer in the first memory.

[0250] In one possible implementation, taking the first sub-image as having multiple image layers including a base layer and a first enhancement layer as an example, the first memory may include a first storage space and a second storage space. The first storage space is used to store the bitstream of the base layer, and the second storage space is used to store the bitstream of the first enhancement layer. After obtaining the bitstream of the base layer of the first sub-image, the encoder can write the bitstream of the base layer of the first sub-image into the first storage space; and after obtaining the bitstream of the first enhancement layer of the first sub-image, it can write the bitstream of the first enhancement layer of the first sub-image into the second storage space.

[0251] Similarly, after obtaining the bitstreams of each image layer of the other sub-images in the first frame image, the encoder can write the bitstreams of each image layer of the other sub-images into their respective storage spaces.

[0252] Optionally, this application does not limit the storage location of the bitstreams of different sub-images in the same storage space of the first memory. In one possible implementation, the encoder can write the bitstreams of adjacent sub-images into adjacent address segments.

[0253] Furthermore, whenever the number of sub-images corresponding to the bitstream of the base layer stored in the first storage space reaches a preset first number, the encoder reads the bitstream of the base layer of the first number of sub-images and sends it to the decoder; similarly, whenever the number of sub-images corresponding to the bitstream of the first enhancement layer stored in the second storage space reaches the first number, the encoder reads the bitstream of the first enhancement layer of the first number of sub-images and sends it to the decoder.

[0254] In a Wi-Fi network, the encoding end needs to preempt the channel for data transmission before sending data to the decoding end each time. However, preemption of the channel can fail, or the success rate of preemption is relatively limited. Therefore, in this application, the bitstream of the same image layer of each sub-image is stored in off-chip memory in sequence. After accumulating a certain amount of data, it is sent to the decoding end. This reduces the number of times the channel is preempted, thereby improving the efficiency of data transmission.

[0255] For example, taking the first sub-image as an image block, the first quantity can be a slice or a frame.

[0256] Please refer to Figure 8Taking a frame of image as an example, divided into m sub-images (e.g., B1, B2...Bm, where m is an integer greater than 2), the encoder performs two layers (e.g., a base layer and a first enhancement layer) encoding on each of the m sub-images. The DRAM includes a first storage space and a second storage space. The first storage space is used to store the bitstream of the base layer, and the second storage space is used to store the bitstream of the first enhancement layer. Figure 8 A schematic diagram of the storage space of DRAM provided in an embodiment of this application is shown.

[0257] like Figure 8 As shown, the encoder sequentially writes the bitstream of the base layer of B1 (i.e., B1-0), the bitstream of the base layer of B2 (i.e., B2-0), ... the bitstream of the base layer of Bm (i.e., Bm-0) into adjacent address segments in the first memory space (e.g., writing the bitstream into the DRAM each time a bitstream is obtained). For example, B1-0 is written into address segment 1, B2-0 is written into address segment 2, and the end address of address segment 1 is adjacent to the start address of address segment 2.

[0258] Similarly, the encoder sequentially writes the bitstream of the first enhancement layer (i.e., B1-1) of B1, the bitstream of the first enhancement layer (i.e., B2-1) of B2, ..., the bitstream of the first enhancement layer (i.e., Bm-1) of Bm into adjacent address segments in the second storage space (e.g., writing the bitstream into the DRAM each time a bitstream is obtained). For example, B1-1 is written into address segment 1', B2-1 is written into address segment 2', and the end address of address segment 1' is adjacent to the start address of address segment 2'.

[0259] Optionally, if a preset transmission condition is met when the bitstreams of n sub-images are stored in the first and second storage spaces (n being an integer less than m), the encoder reads the bitstreams of the base layers of the currently stored n sub-images and the bitstreams of the first enhancement layers of the n sub-images from the DRAM and sends them to the decoding end; then, the bitstreams of the base layers Bn+1, Bn+2...Bm are sequentially written into the first storage space, and the bitstreams of the first enhancement layers Bn+1, Bn+2...Bm are sequentially written into the second storage space, until the above transmission condition is met again, the encoder reads the bitstreams of the base layers of all the sub-images stored this time and the bitstreams of the first enhancement layers of all the sub-images stored this time from the DRAM again and sends them to the decoding end; and so on, until all the bitstreams of m sub-images are sent to the decoding end.

[0260] In one possible implementation, the above-mentioned transmission condition may be that the number of sub-images corresponding to the bitstream stored in the first storage space and the second storage space reaches a preset first number, i.e., n; or, the above-mentioned transmission condition may be that the amount of data of the bitstream stored in the first storage space and the second storage space reaches a preset first amount of data.

[0261] Optionally, if the bitstreams stored in the first and second storage spaces do not meet the above transmission conditions, but all bitstreams obtained from encoding a frame of image have been written into DRAM, then the encoder reads the bitstreams of the base layer of all sub-images stored in this instance and the bitstreams of the first enhancement layer of all sub-images stored in this instance from DRAM, and sends them to the decoding end. Thus, the image encoding device completes the layered encoding and transmission of the entire frame of image.

[0262] Optionally, the method 400 may further include: the encoder may further write the reconstructed images of each image layer of the first sub-image into the off-chip memory to generate an inter-frame reference image of the second frame image; or, the reconstructed images of each image layer of the first sub-image may be processed by loop filtering and then written into the off-chip memory to improve image quality. The second frame image is located after the first frame image in the time-frequency frame sequence.

[0263] Please refer to Figure 9 Taking a sub-image of the entire frame as an example, which is encoded in 3 layers, Figure 9 This diagram illustrates an exemplary encoding process of the encoding end provided in an embodiment of this application. It should be noted that, as... Figure 9 As shown, the encoding process at the encoding end includes the following steps. It should be noted that the steps listed below can be executed in various orders and / or occur simultaneously, and are not limited to... Figure 9 The execution order is shown.

[0264] (1) The encoder reads sub-image 1 from the first frame image from the external memory of the SOC and stores it in the on-chip memory of the SOC.

[0265] It should be noted that, for the sake of clarity, Figure 9 Each black box represents a block written to the chip's internal memory.

[0266] (2) The encoder reads the inter-frame reference image of sub-image 1 from the off-chip memory.

[0267] (3) The encoder reads the inter-frame reference image from the on-chip memory.

[0268] (4) The encoder performs inter-frame prediction on the inter-frame reference image to obtain the predicted image 1.

[0269] (5) The encoder reads the sub-image 1 from the on-chip memory.

[0270] (6) The encoder performs residual calculation on the sub-image 1 and the predicted image 1 to obtain the residual image 1.

[0271] (7) The encoder transforms and quantizes the residual image 1 to obtain the processed image 1.

[0272] (8) The encoder performs entropy coding on the processed image 1 to obtain the bitstream of the basic layer of sub-image 1.

[0273] (9) The encoder writes the bitstream of the basic layer of the sub-image 1 into the off-chip memory.

[0274] (10) The encoder performs inverse transformation and inverse quantization on the processed image 1 to obtain the reconstructed image of the residual image 1.

[0275] (11) The encoder merges the predicted image 1 and the reconstructed image of the residual image 1 to obtain the reconstructed image of the basic layer of the sub-image 1.

[0276] (12) The encoder directly writes the reconstructed image of the basic layer of the sub-image 1 into the on-chip memory.

[0277] (13) The encoder reads the reconstructed image of the basic layer of the sub-image 1 from the on-chip memory.

[0278] (14) The encoder performs inter-layer prediction on the reconstructed image of the basic layer of the sub-image 1 to obtain the predicted image 2.

[0279] (15) The encoder reads the sub-image 1 from the on-chip memory.

[0280] (16) The encoder performs residual calculation on the sub-image 1 and the predicted image 2 to obtain the residual image 2.

[0281] (17) The encoder transforms and quantizes the residual image 2 to obtain the processed image 2.

[0282] (18) The encoder performs entropy coding on the processed image 2 to obtain the bitstream of the enhancement layer 1 of the sub-image 1.

[0283] (19) The encoder writes the bitstream of the enhancement layer 1 of the sub-image 1 into the off-chip memory.

[0284] (20) The encoder performs inverse transformation and inverse quantization on the processed image 2 to obtain the reconstructed image of the residual image 2.

[0285] (21) The encoder merges the predicted image 2 and the reconstructed image of the residual image 2 to obtain the reconstructed image of the enhancement layer 1 of the sub-image 1.

[0286] (22) The encoder directly writes the reconstructed image of the enhancement layer 1 of the sub-image 1 into the on-chip memory.

[0287] (23) The encoder reads the reconstructed image of the enhancement layer 1 of the sub-image 1 from the on-chip memory.

[0288] (24) The encoder performs inter-layer prediction on the reconstructed image of the enhancement layer 1 of the sub-image 1 to obtain the predicted image 3.

[0289] (25) The encoder reads the sub-image 1 from the on-chip memory.

[0290] (26) The encoder performs residual calculation on the sub-image 1 and the predicted image 3 to obtain the residual image 3.

[0291] (27) The encoder transforms and quantizes the residual image 3 to obtain the processed image 3.

[0292] (28) The encoder performs entropy coding on the processed image 3 to obtain the bitstream of the enhancement layer 2 of the sub-image 1.

[0293] (29) The encoder writes the bitstream of the enhancement layer 2 of the sub-image 1 into the off-chip memory.

[0294] (30) The encoder performs inverse transformation and inverse quantization on the processed image 3 to obtain the reconstructed image of the residual image 3.

[0295] (31) The encoder merges the predicted image 3 and the reconstructed image of the residual image 3 to obtain the reconstructed image of the enhancement layer 2 of the sub-image 1.

[0296] (32) The encoder directly writes the reconstructed image of the enhancement layer 2 of the sub-image 1 into the on-chip memory.

[0297] Optionally, this process may also include the following steps.

[0298] (33) The encoder uses loop filtering to process the reconstructed image of the basic layer of the sub-image 1 to obtain the filtered image 1.

[0299] (34) The encoder writes the filtered image 1 into the off-chip memory.

[0300] (35) The encoder performs loop filtering on the reconstructed image of the enhancement layer 1 of the sub-image 1 to obtain the filtered image 2.

[0301] (36) The encoder writes the filtered image 2 into the off-chip memory.

[0302] (37) The encoder performs loop filtering on the reconstructed image of the enhancement layer 1 of the sub-image 1 to obtain the filtered image 3.

[0303] (38) The encoder writes the filtered image 3 into the off-chip memory.

[0304] Optionally, the filtered image 1, filtered image 2, and filtered image 3 can be used to encode inter-frame reference images for the next frame.

[0305] It should be noted that, Figure 9 For details on the content and corresponding beneficial effects not described in the above, please refer to the relevant introduction in Method 400 above, which will not be repeated here.

[0306] Please refer to Figure 10 , Figure 10 A schematic flowchart of an image decoding method 500 provided in an embodiment of this application is shown. Method 500 can be used in the image decoding apparatus 300 described above. Method 500 is described as a series of steps or operations, but it should be understood that method 500 can be performed in various orders and / or occur simultaneously, and is not limited to... Figure 10 The execution order is shown. Figure 10 As shown, method 500 may include the following steps.

[0307] Step 501: The decoder reads the bitstream of multiple image layers of the first sub-image of the first frame image from the first memory and writes the bitstream of the multiple image layers of the first sub-image into the second memory. The first memory is an off-chip memory and the second memory is an on-chip memory. The first frame image includes multiple sub-images, and the multiple sub-images include the first sub-image. The multiple image layers include a base layer and a first enhancement layer. The image quality of the first enhancement layer is higher than that of the base layer.

[0308] In other words, the decoder writes the bitstream of multiple image layers of the first sub-image into the second memory.

[0309] It should be noted that the decoder and the second memory mentioned in step 501 can be located inside the SOC, while the first memory mentioned in step 501 can be located outside the SOC.

[0310] For example, the above SOC can be Figure 3 The SOC 310 shown above can be a decoder that... Figure 3 The decoder 311 shown above, the second memory can be Figure 3 The on-chip memory 312 shown above, the first memory can be... Figure 3 The off-chip memory 320 shown is shown.

[0311] For example, the first memory can be DRAM and the second memory can be SRAM.

[0312] Optionally, the first frame image mentioned above can be any frame in the video frame sequence, and this application embodiment does not limit this.

[0313] Optionally, the first sub-image can be any one of the plurality of sub-images, and this embodiment of the application does not limit this.

[0314] Optionally, the aforementioned sub-image can be, but is not limited to, a block, MB, or LCU within a frame image, or other block-level images; this application embodiment does not limit this. Here, a block-level image refers to an image that does not fully conform to the standard size specified by a block, macroblock, or maximum coding block, but whose size difference from the standard size of a block, macroblock, or maximum coding block is less than a preset size threshold.

[0315] Step 502: The decoder reads the bitstream of the base layer of the first sub-image from the second memory.

[0316] Step 503: The decoder decodes the bitstream of the basic layer of the first sub-image to obtain a first decoded image, which is the decoded image of the basic layer of the first sub-image.

[0317] Step 504: The decoder writes the first decoded image into the second memory.

[0318] In one possible implementation, the decoder can directly write the first decoded image into the on-chip memory. That is, the first decoded image is an image that has not undergone loop filtering.

[0319] Step 505: The decoder reads the bitstream of the first enhancement layer of the first sub-image and the first decoded image from the second memory.

[0320] Step 506: Based on the first decoded image, the decoder decodes the bitstream of the first enhancement layer of the first sub-image to obtain a second decoded image, which is the decoded image of the first enhancement layer of the first sub-image.

[0321] Optionally, after obtaining the second decoded image, the method 500 further includes: the decoder writing the second decoded image into the second memory, and the second decoded image overwriting the first decoded image.

[0322] Optionally, the plurality of image layers further includes a second enhancement layer, the image quality of which is higher than that of the first enhancement layer. The method 500 further includes: the decoder reading the bitstream of the second enhancement layer of the first sub-image and the second decoded image from the second memory; the decoder decoding the bitstream of the second enhancement layer of the first sub-image based on the second decoded image to obtain a third decoded image, the third decoded image being the decoded image of the second enhancement layer of the first sub-image; and the decoder writing the third decoded image into the second memory, wherein the third decoded image overwrites the second decoded image.

[0323] Optionally, the method further includes: the decoder writing the decoded image of the target image layer of the first sub-image into the first memory, wherein the target image layer is the image layer with the highest image quality among the plurality of image layers.

[0324] Furthermore, whenever the number of sub-images corresponding to the decoded images stored in the first memory reaches a preset first number, the decoder reads the decoded image of the first enhancement layer of the first number of sub-images and sends it to the display device.

[0325] Optionally, the plurality of sub-images further includes a second sub-image, which is adjacent to the first sub-image in the decoding order. The method 500 further includes: after decoding the bitstream of the plurality of image layers of the first sub-image, the decoder decodes the bitstream of the plurality of image layers of the second sub-image; or, while decoding the bitstream of the first enhancement layer of the first sub-image, the decoder decodes the bitstream of the base layer of the second sub-image.

[0326] Optionally, the decoder can also write the decoded images of each image layer of the first sub-image into the off-chip memory to generate an inter-frame reference image for the second frame; or, it can apply loop filtering to the decoded images of each image layer of the first sub-image before writing them into the off-chip memory to improve image quality. The second frame image is located after the first frame image in the time-frequency frame sequence.

[0327] It should be noted that, since the decoding process is the reverse of the encoding process, therefore, Figure 10 For details not covered in the text and the beneficial effects, please refer to [the relevant source]. Figure 4 This will not be elaborated upon here.

[0328] At this point, the image decoding device has completed the layered decoding and display of the entire frame image bitstream.

[0329] The above combination Figures 4 to 10 The encoding and decoding methods provided in the embodiments of this application have been introduced. The encoding and decoding methods provided in the embodiments of this application will be introduced below.

[0330] Please refer to Figure 11 , Figure 11 A schematic diagram of the encoder 600 provided in an embodiment of this application is shown. The encoder 600 can be used as an encoding device at the encoding end. Figure 11 As shown, encoder 600 may include: read / write control module 601 and encoding module 602.

[0331] The read / write control module 601 is used to read a first sub-image of a first frame image from the first memory and store the first sub-image into the second memory. The first memory is an off-chip memory and the second memory is an on-chip memory. The first frame image includes multiple sub-images, and the multiple sub-images include the first sub-image. The first sub-image is read from the second memory.

[0332] The encoding module 602 is used to encode the basic layer of the first sub-image based on the first sub-image to obtain the bitstream of the basic layer of the first sub-image and a first reconstructed image, wherein the first reconstructed image is the reconstructed image of the basic layer of the first sub-image.

[0333] The read / write control module 601 is also used to write the first reconstructed image into the second memory; and to read the first reconstructed image and the first sub-image from the second memory.

[0334] The encoding module 602 is further configured to encode the first enhancement layer of the first sub-image based on the first reconstructed image and the first sub-image to obtain the bitstream of the first enhancement layer of the first sub-image and the second reconstructed image, wherein the second reconstructed image is the reconstructed image of the first enhancement layer of the first sub-image, and the image quality of the first enhancement layer is higher than that of the base layer.

[0335] In one possible implementation, the encoding module 602 is specifically used to directly write the second reconstructed image into the second memory.

[0336] In one possible implementation, the read / write control module 601 is further configured to write the second reconstructed image into the second memory after obtaining the second reconstructed image, and the second reconstructed image overwrites the first reconstructed image.

[0337] In one possible implementation, the read / write control module 601 is further configured to read the second reconstructed image and the first sub-image from the second memory; the encoding module 602 is further configured to encode the second enhancement layer of the first sub-image based on the second reconstructed image and the first sub-image to obtain the bitstream of the second enhancement layer of the first sub-image and a third reconstructed image, wherein the third reconstructed image is a reconstructed image of the second enhancement layer of the first sub-image, and the image quality of the second enhancement layer is higher than that of the first enhancement layer; the read / write control module 601 is further configured to write the third reconstructed image to the second memory, and the third reconstructed image covers the second reconstructed image.

[0338] In one possible implementation, the first memory includes a first storage space and a second storage space. The first storage space is used to store the bitstream of the base layer, and the second storage space is used to store the bitstream of the first enhancement layer. The read / write control module 601 is also used to write the bitstream of the base layer of the first sub-image into the first storage space and to write the bitstream of the first enhancement layer of the first sub-image into the second storage space.

[0339] In one possible implementation, the encoder 600 may further include a transmitting module 603; the read / write control module 601 is further configured to read the bitstream of the basic layer of the first number of sub-images whenever the number of sub-images corresponding to the bitstream of the basic layer stored in the first storage space reaches a preset first number; the transmitting module 603 is configured to send the bitstream of the basic layer of the first number of sub-images to the decoding end; the read / write control module 601 is further configured to read the bitstream of the first enhancement layer of the first number of sub-images whenever the number of sub-images corresponding to the bitstream of the first enhancement layer stored in the second storage space reaches the first number; the transmitting module 603 is further configured to send the bitstream of the first enhancement layer of the first number of sub-images to the decoding end.

[0340] In one possible implementation, the plurality of sub-images further includes a second sub-image, and the second sub-image is adjacent to the encoding order of the first sub-image. The encoding module 602 is further configured to encode the plurality of image layers of the second sub-image after completing the encoding of the plurality of image layers of the first sub-image, the plurality of image layers including the base layer and the first enhancement layer; or, the base layer of the second sub-image is encoded while the first enhancement layer of the first sub-image is encoded.

[0341] In one possible implementation, the sub-image is a block, a macroblock, or a maximum coded block.

[0342] In one possible implementation, the second memory is a static random access memory (SRAM), and the first memory is a dynamic random access memory (DRAM).

[0343] Encoder 600 can be used to perform Figure 4 The technical solutions of the method embodiments shown are similar in implementation principle and technical effect, and will not be described again here.

[0344] Please refer to Figure 12 , Figure 12 A schematic diagram of the decoder 700 provided in an embodiment of this application is shown. The decoder 700 can be used as a decoding device at the decoding end. Figure 12 As shown, the decoder 700 may include a read / write control module 701 and a decoding module 702.

[0345] The read / write control module 701 is used to read the bitstream of multiple image layers of the first sub-image of the first frame image from the first memory, and write the bitstream of the multiple image layers of the first sub-image into the second memory. The first memory is an off-chip memory, and the second memory is an on-chip memory. The first frame image includes multiple sub-images, and the multiple sub-images include the first sub-image. The multiple image layers include a base layer and a first enhancement layer. The image quality of the first enhancement layer is higher than that of the base layer.

[0346] The decoding module 702 is used to decode the bitstream of the basic layer of the first sub-image to obtain a first decoded image, which is the decoded image of the basic layer of the first sub-image.

[0347] The read / write control module 701 is also used to write the first decoded image into the second memory; and to read the bitstream of the first enhancement layer of the first sub-image and the first decoded image from the second memory.

[0348] The decoding module 702 is also used to decode the bitstream of the first enhancement layer of the first sub-image based on the first decoded image to obtain a second decoded image, wherein the second decoded image is the decoded image of the first enhancement layer of the first sub-image.

[0349] In one possible implementation, the read / write control module 701 is specifically used to directly write the first decoded image into the second memory.

[0350] In one possible implementation, the read / write control module 701 is further configured to write the second decoded image into the second memory after obtaining the second decoded image, and the second decoded image overwrites the first decoded image.

[0351] In one possible implementation, the plurality of image layers further includes a second enhancement layer, the image quality of which is higher than that of the first enhancement layer. The read / write control module 701 is further configured to read the bitstream of the second enhancement layer of the first sub-image and the second decoded image from the second memory. The decoding module 702 is further configured to decode the bitstream of the second enhancement layer of the first sub-image based on the second decoded image to obtain a third decoded image, which is the decoded image of the second enhancement layer of the first sub-image. The read / write control module 701 is further configured to write the third decoded image into the second memory, and the third decoded image overwrites the second decoded image.

[0352] In one possible implementation, the read / write control module 701 is further used to write the decoded image of the target image layer of the first sub-image into the first memory, wherein the target image layer is the image layer with the highest image quality among the plurality of image layers.

[0353] In one possible implementation, the decoder 700 may further include a sending module 703, and the read / write control module 701 is further configured to read the decoded image of the first enhancement layer of the first number of sub-images when the number of sub-images corresponding to the decoded images stored in the first memory reaches a preset first number; the sending module 703 is configured to send the decoded image of the first enhancement layer of the first number of sub-images to the display device.

[0354] In one possible implementation, the plurality of sub-images further includes a second sub-image, which is adjacent to the first sub-image in the decoding order. The decoding module 702 is further configured to decode the bitstream of the plurality of image layers of the second sub-image after decoding the bitstream of the plurality of image layers of the first sub-image; or, the decoding module 702 is further configured to decode the bitstream of the base layer of the second sub-image while decoding the bitstream of the first enhancement layer of the first sub-image.

[0355] In one possible implementation, the sub-image is a block, a macroblock, or a maximum coded block.

[0356] In one possible implementation, the second memory is a static random access memory (SRAM), and the first memory is a dynamic random access memory (DRAM).

[0357] Decoder 700 can be used to execute Figure 10 The technical solutions of the method embodiments shown are similar in principle and in effect, and will not be described again here.

[0358] This application also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the above-described related method steps to implement the image encoding method and / or image decoding method in the above embodiments.

[0359] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the image encoding method and / or image decoding method described in the above embodiments.

[0360] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0361] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0362] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0363] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0364] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0365] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0366] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image encoding method, characterized in that, include: The first sub-image of the first frame image is read from the first memory and stored in the second memory. The first memory is an off-chip memory and the second memory is an on-chip memory. The first frame image includes multiple sub-images, and the multiple sub-images include the first sub-image. Based on the first sub-image, the basic layer of the first sub-image is encoded to obtain the bitstream of the basic layer of the first sub-image and the first reconstructed image, wherein the first reconstructed image is the reconstructed image of the basic layer of the first sub-image. Write the first reconstructed image into the second memory; Read the first reconstructed image and the first sub-image from the second memory; Based on the first reconstructed image and the first sub-image, the first enhancement layer of the first sub-image is encoded to obtain the bitstream of the first enhancement layer of the first sub-image and the second reconstructed image. The second reconstructed image is the reconstructed image of the first enhancement layer of the first sub-image, and the image quality of the first enhancement layer is higher than that of the base layer.

2. The method according to claim 1, characterized in that, After obtaining the second reconstructed image, the method further includes: The second reconstructed image is written to the second memory, and the second reconstructed image overwrites the first reconstructed image.

3. The method according to claim 2, characterized in that, The method further includes: Read the second reconstructed image and the first sub-image from the second memory; Based on the second reconstructed image and the first sub-image, the second enhancement layer of the first sub-image is encoded to obtain the bitstream of the second enhancement layer of the first sub-image and the third reconstructed image. The third reconstructed image is the reconstructed image of the second enhancement layer of the first sub-image. The image quality of the second enhancement layer is higher than that of the first enhancement layer. The third reconstructed image is written into the second memory, and the third reconstructed image overwrites the second reconstructed image.

4. The method according to any one of claims 1-3, characterized in that, The first memory includes a first storage space and a second storage space, the first storage space being used to store the bitstream of the base layer, and the second storage space being used to store the bitstream of the first enhancement layer. The method further includes: Write the bitstream of the basic layer of the first sub-image into the first storage space; Write the bitstream of the first enhancement layer of the first sub-image into the second storage space.

5. The method according to claim 4, characterized in that, The method further includes: Whenever the number of sub-images corresponding to the bitstream of the base layer stored in the first storage space reaches a preset first number, the bitstream of the base layer of the first number of sub-images is read and sent to the decoding end. Whenever the number of sub-images corresponding to the bitstream of the first enhancement layer stored in the second storage space reaches the first number, the bitstream of the first enhancement layer of the first number of sub-images is read and sent to the decoding end.

6. The method according to any one of claims 1-5, characterized in that, The plurality of sub-images further includes a second sub-image, and the second sub-image is adjacent to the first sub-image in encoding order. The method further includes: After encoding multiple image layers of the first sub-image, multiple image layers of the second sub-image are encoded, including the base layer and the first enhancement layer; or, While encoding the first enhancement layer of the first sub-image, the base layer of the second sub-image is encoded.

7. The method according to any one of claims 1-6, characterized in that, The sub-image is a block, a macroblock, or a maximum coded block.

8. The method according to any one of claims 1-7, characterized in that, The second memory is a static random access memory (SRAM), and the first memory is a dynamic random access memory (DRAM).

9. An image decoding method, characterized in that, include: The first frame image reads the bitstream of multiple image layers of the first sub-image of the first frame image from the first memory and writes the bitstream of multiple image layers of the first sub-image into the second memory. The first memory is an off-chip memory and the second memory is an on-chip memory. The first frame image includes multiple sub-images, and the multiple sub-images include the first sub-image. The multiple image layers include a base layer and a first enhancement layer. The image quality of the first enhancement layer is higher than that of the base layer. Decode the bitstream of the basic layer of the first sub-image to obtain a first decoded image, which is the decoded image of the basic layer of the first sub-image. Write the first decoded image into the second memory; Read the bitstream of the first enhancement layer of the first sub-image and the first decoded image from the second memory; Based on the first decoded image, the bitstream of the first enhancement layer of the first sub-image is decoded to obtain a second decoded image, which is the decoded image of the first enhancement layer of the first sub-image.

10. The method according to claim 9, characterized in that, After obtaining the second decoded image, the method further includes: The second decoded image is written into the second memory, and the second decoded image overwrites the first decoded image.

11. The method according to claim 10, characterized in that, The plurality of image layers further includes a second enhancement layer, wherein the image quality of the second enhancement layer is higher than that of the first enhancement layer, and the method further includes: Read the bitstream of the second enhancement layer of the first sub-image and the second decoded image from the second memory; Based on the second decoded image, the bitstream of the second enhancement layer of the first sub-image is decoded to obtain a third decoded image, which is the decoded image of the second enhancement layer of the first sub-image. The third decoded image is written into the second memory, and the third decoded image overwrites the second decoded image.

12. The method according to any one of claims 9-11, characterized in that, The method further includes: The decoded image of the target image layer of the first sub-image is written into the first memory, wherein the target image layer is the image layer with the highest image quality among the plurality of image layers.

13. The method according to claim 12, characterized in that, The method further includes: Whenever the number of sub-images corresponding to the decoded images stored in the first memory reaches a preset first number, the decoded images of the first number of sub-images are read and sent to the display device.

14. The method according to any one of claims 9-13, characterized in that, The plurality of sub-images further includes a second sub-image, the second sub-image being adjacent to the first sub-image in decoding order, and the method further includes: After decoding the bitstreams of multiple image layers of the first sub-image, the bitstreams of multiple image layers of the second sub-image are decoded; or, While decoding the bitstream of the first enhancement layer of the first sub-image, the bitstream of the base layer of the second sub-image is also decoded.

15. The method according to any one of claims 9-14, characterized in that, The sub-image is a block, a macroblock, or a maximum coded block.

16. The method according to any one of claims 9-15, characterized in that, The second memory is a static random access memory (SRAM), and the first memory is a dynamic random access memory (DRAM).

17. An image encoding device, characterized in that, include: The system-on-a-chip (SOC) includes an encoder, a second memory, and a communication interface. The encoder, the second memory, and the communication interface are coupled to each other. The first memory is located outside the SOC and is coupled to the communication interface. The encoder is used for: The first sub-image of the first frame image is read from the first memory and stored in the second memory. The first memory is an off-chip memory and the second memory is an on-chip memory. The first frame image includes multiple sub-images, and the multiple sub-images include the first sub-image. Based on the first sub-image, the basic layer of the first sub-image is encoded to obtain the bitstream of the basic layer of the first sub-image and the first reconstructed image, wherein the first reconstructed image is the reconstructed image of the basic layer of the first sub-image. Write the first reconstructed image into the second memory; Read the first reconstructed image and the first sub-image from the second memory; Based on the first reconstructed image and the first sub-image, the first enhancement layer of the first sub-image is encoded to obtain the bitstream of the first enhancement layer of the first sub-image and the second reconstructed image. The second reconstructed image is the reconstructed image of the first enhancement layer of the first sub-image, and the image quality of the first enhancement layer is higher than that of the base layer.

18. The apparatus according to claim 17, characterized in that, The encoder is also used for: After obtaining the second reconstructed image, the second reconstructed image is written into the second memory, and the second reconstructed image overwrites the first reconstructed image.

19. The apparatus according to claim 18, characterized in that, The encoder is also used for: Read the second reconstructed image and the first sub-image from the second memory; Based on the second reconstructed image and the first sub-image, the second enhancement layer of the first sub-image is encoded to obtain the bitstream of the second enhancement layer of the first sub-image and the third reconstructed image. The third reconstructed image is the reconstructed image of the second enhancement layer of the first sub-image. The image quality of the second enhancement layer is higher than that of the first enhancement layer. The third reconstructed image is written into the second memory, and the third reconstructed image overwrites the second reconstructed image.

20. The apparatus according to any one of claims 17-19, characterized in that, The first memory includes a first storage space and a second storage space. The first storage space is used to store the bitstream of the base layer, and the second storage space is used to store the bitstream of the first enhancement layer. The encoder is further used for: Write the bitstream of the basic layer of the first sub-image into the first storage space; Write the bitstream of the first enhancement layer of the first sub-image into the second storage space.

21. The apparatus according to claim 20, characterized in that, The encoder is also used for: Whenever the number of sub-images corresponding to the bitstream of the base layer stored in the first storage space reaches a preset first number, the bitstream of the base layer of the first number of sub-images is read and sent to the decoding end. Whenever the number of sub-images corresponding to the bitstream of the first enhancement layer stored in the second storage space reaches the first number, the bitstream of the first enhancement layer of the first number of sub-images is read and sent to the decoding end.

22. The apparatus according to any one of claims 17-21, characterized in that, The plurality of sub-images further includes a second sub-image, and the second sub-image is adjacent to the first sub-image in encoding order. The encoder is further configured to: After encoding multiple image layers of the first sub-image, multiple image layers of the second sub-image are encoded, including the base layer and the first enhancement layer. or, While encoding the first enhancement layer of the first sub-image, the base layer of the second sub-image is encoded.

23. The apparatus according to any one of claims 17-22, characterized in that, The sub-image is a block, a macroblock, or a maximum coded block.

24. The apparatus according to any one of claims 17-23, characterized in that, The second memory is a static random access memory (SRAM), and the first memory is a dynamic random access memory (DRAM).

25. An image decoding device, characterized in that, include: The system-on-a-chip (SOC) includes a decoder, a second memory, and a communication interface. The decoder, the second memory, and the communication interface are coupled to each other. The first memory is located outside the SOC and is coupled to the communication interface. The decoder is used for: The first frame image reads the bitstream of multiple image layers of the first sub-image of the first frame image from the first memory and writes the bitstream of multiple image layers of the first sub-image into the second memory. The first memory is an off-chip memory and the second memory is an on-chip memory. The first frame image includes multiple sub-images, and the multiple sub-images include the first sub-image. The multiple image layers include a base layer and a first enhancement layer. The image quality of the first enhancement layer is higher than that of the base layer. Decode the bitstream of the basic layer of the first sub-image to obtain a first decoded image, which is the decoded image of the basic layer of the first sub-image. Write the first decoded image into the second memory; Read the bitstream of the first enhancement layer of the first sub-image and the first decoded image from the second memory; Based on the first decoded image, the bitstream of the first enhancement layer of the first sub-image is decoded to obtain a second decoded image, which is the decoded image of the first enhancement layer of the first sub-image.

26. The apparatus according to claim 25, characterized in that, The decoder is also used for: After obtaining the second decoded image, the second decoded image is written into the second memory, and the second decoded image overwrites the first decoded image.

27. The apparatus according to claim 26, characterized in that, The plurality of image layers further includes a second enhancement layer, the second enhancement layer having image quality higher than that of the first enhancement layer, and the decoder is further configured to: Read the bitstream of the second enhancement layer of the first sub-image and the second decoded image from the second memory; Based on the second decoded image, the bitstream of the second enhancement layer of the first sub-image is decoded to obtain a third decoded image, which is the decoded image of the second enhancement layer of the first sub-image. The third decoded image is written into the second memory, and the third decoded image overwrites the second decoded image.

28. The apparatus according to any one of claims 25-27, characterized in that, The decoder is also used for: The decoded image of the target image layer of the first sub-image is written into the first memory, wherein the target image layer is the image layer with the highest image quality among the plurality of image layers.

29. The apparatus according to claim 28, characterized in that, The decoder is also used for: Whenever the number of sub-images corresponding to the decoded images of the first enhancement layer stored in the first memory reaches a preset first number, the decoded images of the first enhancement layer of the first number of sub-images are read and sent to the display device.

30. The apparatus according to any one of claims 25-29, characterized in that, The plurality of sub-images further includes a second sub-image, the second sub-image being adjacent to the first sub-image in decoding order, and the decoder is further configured to: After decoding the bitstreams of multiple image layers of the first sub-image, the bitstreams of multiple image layers of the second sub-image are decoded; or, While decoding the bitstream of the first enhancement layer of the first sub-image, the bitstream of the base layer of the second sub-image is also decoded.

31. The apparatus according to any one of claims 25-30, characterized in that, The sub-image is a block, a macroblock, or a maximum coded block.

32. The apparatus according to any one of claims 25-31, characterized in that, The second memory is a static random access memory (SRAM), and the first memory is a dynamic random access memory (DRAM).

33. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the method as described in any one of claims 1-8 or 9-16.

34. A computer program product, characterized in that, Includes computer instructions that, when executed on a processor, implement the method as described in any one of claims 1-8 or 9-16.

Images