Image coding method, apparatus, encoder, decoder and system

Abstract

Image encoding and decoding methods, apparatus, encoders, decoders, and systems are disclosed, relating to the multimedia field. The method includes decoding the bitstream of a coding unit to be decoded in an image bitstream, or encoding a coding unit to be encoded in the current frame, while dynamically setting a target bit count, taking into account both the image content of the coding unit and the number of bits of data in the bitstream buffer. At the encoding end, the coding unit is encoded using quantization parameters determined by the target bit count. At the decoding end, the bitstream of the coding unit is decoded according to the quantization parameters determined by the target bit count. This reduces the number of encoded bits after encoding the image while ensuring the quality of the reconstructed image.

Description

[0001] This application is a divisional application. The original application has the application number 202211696765.4 and the original application date is December 28, 2022. The entire contents of the original application are incorporated herein by reference.

[0002] This application claims priority to Chinese patent application filed on September 8, 2022, with application number 202211097577.X, entitled "Clamping Method for Target Bit Count", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of multimedia, and more particularly to an image encoding and decoding method, apparatus, encoder, decoder and system. Background Technology

[0004] Currently, encoders perform encoding operations such as prediction, quantization, and entropy coding on image frames to obtain a bitstream. Decoders then perform decoding operations such as entropy decoding, dequantization, and prediction reconstruction on the bitstream to obtain the reconstructed image of the image frame. However, a larger quantization parameter value results in less effective information from the image frame contained in the bitstream, leading to poorer reconstructed image quality; conversely, a smaller quantization parameter value results in higher reconstructed image quality, but also more redundant information from the image frame contained in the bitstream, resulting in a larger bitstream size. Therefore, determining the quantization parameters used for image encoding and decoding, while ensuring the quality of the reconstructed image, and reducing the number of encoded bits after encoding, is a pressing issue that needs to be addressed. Summary of the Invention

[0005] This application provides an image encoding and decoding method, apparatus, encoder, decoder, and system, thereby reducing the number of encoded bits after encoding the image while ensuring the quality of the reconstructed image by reasonably determining the quantization parameters used for encoding and decoding the image.

[0006] Firstly, an image decoding method is provided. This method includes: when decoding the bitstream of a coding unit to be decoded in an image bitstream, determining the target number of bits for the coding unit based on the image content of the coding unit and the number of bits in the bitstream buffer, and decoding the bitstream of the coding unit based on quantization parameters determined by the target number of bits. The image content is used to indicate the complexity of different pixel regions within the coding unit. The bitstream buffer is used to store the number of encoded bits after decoding one or more coding units. The target number of bits for the coding unit indicates the expected number of bits after lossy encoding of the coding unit with reference to the image content of the coding unit.

[0007] Higher complexity of a coding unit indicates more information in the image, meaning less repetitive information; conversely, lower complexity indicates less information in the image, meaning more repetitive information. When encoding a coding unit, both the image content of the coding unit and the number of bits in the bitstream buffer are considered. For coding units with lower complexity, a smaller target bit count is preferred; for coding units with higher complexity, a larger target bit count is preferred. That is, considering the complexity of the image content expressed by the coding unit and the number of bits of the already encoded coding units in the bitstream buffer, the goal is to minimize the number of encoded bits after encoding the image, while ensuring the quality of the reconstructed image. The reverse process of encoding a coding unit—decoding the bitstream of the coding unit—also considers both the image content of the coding unit and the number of bits in the bitstream buffer. By dynamically setting the target bit count and using the quantization parameters determined by the target bit count to decode the bitstream of the coding unit, the accuracy of bit rate control is improved, thereby minimizing the number of encoded bits after encoding the image while ensuring the quality of the reconstructed image.

[0008] For example, when encoding video using a constant bitrate control strategy, if the number of bits in the bitstream of the encoding unit is small, but the image content of the unit to be encoded is complex, the number of bits in the bitstream of the encoding unit can be reasonably increased while ensuring a constant bitrate, and the quantization parameter value can be smaller to improve the quality of the reconstructed image. Conversely, if the number of bits in the bitstream of the encoding unit is large, but the image content of the unit to be encoded is simple, the quantization parameter value can be larger while ensuring a constant bitrate and the quality of the reconstructed image, thus reasonably reducing the number of bits in the bitstream of the encoding unit.

[0009] In conjunction with the first aspect, in another possible implementation, the method further includes: after determining the quantization parameters, decoding the bit stream of the coding unit according to the quantization parameters to obtain the reconstructed image of the coding unit.

[0010] Secondly, an image coding method is provided, comprising: when encoding a coding unit to be encoded in the current frame, determining the target bit count of the coding unit based on the image content of the coding unit and the number of bits of data in the bitstream buffer, and encoding the coding unit based on quantization parameters determined by the target bit count. The image content is used to indicate the complexity of different pixel regions in the coding unit. The bitstream buffer is used to store the bitstream or a portion of the bitstream of one or more coding units. The target bit count of the coding unit indicates the expected bit count after lossy encoding of the coding unit with reference to the image content of the coding unit.

[0011] Higher complexity of a coding unit indicates more information in the image, meaning less repetitive information; conversely, lower complexity indicates less information in the image, meaning more repetitive information. When encoding a coding unit, both the image content of the unit and the number of bits in the bitstream buffer are considered. For lower complexity coding units, a smaller target bit count is preferred, while for higher complexity units, a larger target bit count is preferred. By dynamically setting the target bit count and using the quantization parameters determined by the target bit count to encode the unit, the accuracy of rate control is improved, minimizing the number of encoded bits while ensuring the quality of the reconstructed image.

[0012] In one possible implementation, determining the target number of bits for a coding unit based on its image content and the number of bits in the bitstream buffer includes: determining the lossless bit data of the coding unit based on its image content, and determining the information content of the coding unit based on the number of bits in the bitstream buffer. Then, the target number of bits is determined based on the lossless bit count and the information content of the coding unit. Since the lossless bit data indicates the expected number of bits after lossless encoding of the coding unit, i.e., the lossless bit data characterizes the expected number of bits under an encoding method that can fully express the information of the coding unit, and the information content indicates the complexity of the image content expressed by the coding unit in the current frame, determining the target number of bits for the coding unit by measuring the lossless bit data of the coding unit based on the information content—that is, reducing the number of encoded bits after encoding the image while fully preserving the information content—improves the accuracy of determining the target number of bits for the coding unit.

[0013] In another possible implementation, the information content of a coding unit in the current frame is determined based on the number of bits of data in the bitstream buffer. This includes: determining the lossy bit count of the coding unit based on the number of bits of data in the bitstream buffer, and determining the information content based on the lossy bit count and the average lossless bit count of the coding unit. The lossy bit count of the coding unit indicates the expected number of bits after lossy encoding of the coding unit without referencing the image content of the coding unit. The average lossless bit count indicates the average expected number of bits after lossless encoding of each coding unit in the current frame. The average expected bit count characterizes the complexity of the image content expressed in the current frame, and the ratio of the lossy bit count to the average expected bit count quantifies the complexity of the image content expressed by the coding unit in the current frame, improving the accuracy of rate control.

[0014] In another possible implementation, the quantization parameters are determined based on the target number of bits in the coding unit. This includes clamping the target number of bits in the coding unit based on at least one of the lossless number of bits, the lossy number of bits in the coding unit, and the full size of the bitstream buffer, to obtain a clamping value for the target number of bits. The quantization parameters are then determined based on the lossless number of bits in the coding unit and the clamping value for the target number of bits. The full size of the bitstream buffer indicates the ratio of the number of bits of data in the bitstream buffer to the storage capacity of the bitstream buffer. Therefore, clamping the target number of bits in the coding unit further improves the accuracy of rate control.

[0015] The image content includes the complexity level of the coding unit. For example, the complexity level of the coding unit includes at least one of the luminance complexity level and the chrominance complexity level.

[0016] Thirdly, an image encoding / decoding apparatus is provided, the apparatus comprising modules for performing the first aspect or any possible design of the first aspect, and modules for performing the second aspect or any possible design of the second aspect.

[0017] Fourthly, an encoder is provided, the encoder including at least one processor and a memory, wherein the memory is used to store a computer program such that when the computer program is executed by at least one processor, it implements the method described in the second aspect or any possible design of the second aspect.

[0018] Fifthly, a decoder is provided, the decoder including at least one processor and a memory, wherein the memory is used to store a computer program such that when the computer program is executed by at least one processor, it implements the method described in the first aspect or any possible design of the first aspect.

[0019] In a sixth aspect, a coding and decoding system is provided, the coding and decoding system comprising an encoder as described in the fourth aspect and a decoder as described in the fifth aspect.

[0020] In a seventh aspect, a chip is provided, comprising: a processor and a power supply circuit; wherein the power supply circuit is used to supply power to the processor; the processor is used to perform operational steps of the method in the first aspect or any possible implementation of the first aspect, and to perform operational steps of the method in the second aspect or any possible implementation of the second aspect.

[0021] Eighthly, a computer-readable storage medium is provided, comprising: computer software instructions; when the computer software instructions are executed in a computing device, causing the computing device to perform operational steps of the method in the first aspect or any possible implementation thereof, and to perform operational steps of the method as described in the second aspect or any possible implementation thereof.

[0022] Ninth aspect, a computer program product is provided, which, when run on a computer, causes a computing device to perform operational steps of the method in the first aspect or any possible implementation of the first aspect, and to perform operational steps of the method as described in the second aspect or any possible implementation of the second aspect.

[0023] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0024] Figure 1 A schematic diagram of the structure of an encoding / decoding system provided in this application;

[0025] Figure 2 A schematic diagram of a coding / decoding system provided in this application;

[0026] Figure 3 A schematic diagram of the structure of an encoder and decoder provided in this application;

[0027] Figure 4 A flowchart illustrating an image encoding / decoding method provided in this application;

[0028] Figure 5 A flowchart illustrating an image encoding method provided in this application;

[0029] Figure 6 A flowchart illustrating an image decoding method provided in this application;

[0030] Figure 7 A schematic diagram of a clamping method for a target number of bits provided in this application;

[0031] Figure 8 A schematic diagram of the structure of an encoding / decoding device provided in this application;

[0032] Figure 9 This is a schematic diagram of the structure of an encoding / decoding system provided in this application. Detailed Implementation

[0033] The technical solutions involved in this application may be applied not only to video coding standards (such as H.264, HEVC, etc.), but also to future video coding standards (such as H.266). The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. A brief introduction to some concepts that may be involved in this application is given below.

[0034] Video consists of multiple consecutive images. When these multiple consecutive images change at more than 24 frames per second, according to the principle of visual persistence, the human eye cannot distinguish a single static image. Thus, multiple images that appear smooth and continuous constitute video.

[0035] Video coding refers to the processing of a sequence of images that forms a video or video sequence. In the field of video coding, the terms "picture," "frame," or "image" can be used synonymously. As used herein, video coding refers to either video encoding or video decoding. Video coding is performed on the source side and typically involves processing (e.g., compressing) the original video images to reduce the amount of data required to represent them, thereby enabling more efficient storage and / or transmission, while maintaining a certain image quality. Video decoding is performed on the destination side and typically involves the inverse processing relative to the encoder to reconstruct the video images. The "encoding" of video images in the embodiments should be understood as involving the "encoding" or "decoding" of a video sequence. The combination of encoding and decoding is also called encoding and decoding (encoding and decoding). Video coding can also be called image coding or image compression. Image decoding refers to the inverse process of image coding.

[0036] A video sequence consists of a series of pictures, which are further divided into slices, and slices into blocks. Video coding is performed on a block-by-block basis. In some newer video coding standards, the concept of a block has been further expanded. For example, the H.264 standard uses macroblocks (MBs), which can be further divided into multiple prediction blocks (partitions) for predictive coding. The High Efficiency Video Coding (HEVC) standard uses basic concepts such as coding units (CUs), prediction units (PUs), and transform units (TUs) to functionally divide various block units, and employs a novel tree-based structure for description. For instance, a CU can be divided into smaller CUs using a quadtree, and these smaller CUs can be further divided, forming a quadtree structure. The CU is the basic unit for partitioning and encoding the image. Similar tree structures exist for PUs and TUs. A PU corresponds to a prediction block and is the basic unit for predictive coding. CUs are further divided into multiple PUs according to partitioning patterns. TU can correspond to transform block, which is the basic unit for transforming prediction residuals. However, whether it is CU, PU or TU, they all essentially belong to the concept of block (or coding unit).

[0037] For example, in HEVC, the CTU is split into multiple CUs using a quadtree structure represented as a coding tree. At the CU level, a decision is made on whether to use inter-picture (temporal) or intra-picture (spatial) prediction to encode picture regions. Each CU can be further split into one, two, or four PUs based on the PU splitting type. The same prediction process is applied within a PU, and relevant information is transmitted to the decoder based on the PU. After obtaining residual blocks by applying the prediction process based on the PU splitting type, the CU can be segmented into transform units (TUs) according to other quadtree structures similar to the coding tree used for CUs. In the latest developments in video compression technology, quadtree and binary tree (QTBT) frame segmentation is used to divide coding blocks. In the QTBT block structure, CUs can be square or rectangular in shape.

[0038] In this paper, for ease of description and understanding, the coding unit to be encoded in the current encoded image is referred to as the current block. For example, in encoding, it refers to the block currently being encoded; in decoding, it refers to the block currently being decoded. The decoded coding unit in the reference image used to predict the current block is called the reference block. That is, the reference block is the block that provides a reference signal for the current block, where the reference signal represents the pixel value within the coding unit. The block in the reference image that provides the prediction signal for the current block can be called the prediction block, where the prediction signal represents the pixel value, sampled value, or sampled signal within the prediction block. For example, after traversing multiple reference blocks, an optimal reference block is found. This optimal reference block will provide prediction for the current block; this block is called the prediction block.

[0039] Lossless video coding refers to the ability to reconstruct the original video images, meaning that the reconstructed video images have the same quality as the original video images (assuming no transmission loss or other data loss during storage or transmission).

[0040] Lossy video coding refers to reducing the number of bits required to represent video images by performing further compression, such as quantization, while the decoder cannot fully reconstruct the video images. In other words, the quality of the reconstructed video images is lower or worse than that of the original video images.

[0041] Bitstream (data rate) refers to the binary stream generated after encoding an image or video. Also known as bitrate or bit depth, it represents the number of bits transmitted per unit of time. It's a crucial part of image quality control in image encoding. For images of the same resolution, a higher bitstream results in a lower compression ratio and better image quality.

[0042] Rate control refers to the adjustment of the bit rate during encoding and decoding, and is abbreviated as code control below. Rate control modes include constant bit rate (CBR) and variable bit rate (VBR).

[0043] Constant Bit Rate (CBR) means ensuring a stable bit rate over a given period of time.

[0044] Dynamic bit rate (VBR) allows bit rate fluctuations within a bit rate statistical period, thereby ensuring stable image quality after encoding.

[0045] Quantization refers to the process of mapping continuous values ​​of a signal into multiple discrete amplitude values.

[0046] The quantization parameter (QP) is used during the encoding process to quantize the residual values ​​from the prediction operation or the coefficients from the transform operation. During decoding, the syntax elements are dequantized to obtain the residual values ​​or coefficients. The quantization parameter is a parameter used in the quantization process; generally, a larger quantization parameter value results in a more pronounced quantization, but also lower image quality and bitrate after reconstruction; conversely, a smaller quantization parameter value results in better image quality and higher bitrate after reconstruction.

[0047] Bitstream buffer fullness refers to the ratio of the number of bits of data in the bitstream buffer to the buffer's storage capacity. At the encoding end, the number of bits of data in the bitstream buffer includes the number of encoded bits in the encoding unit. At the decoding end, the number of bits of data in the bitstream buffer includes the number of decoded bits in the encoding unit.

[0048] Clamping refers to the operation of restricting a certain value to a specified range.

[0049] The embodiments of this application will now be described with reference to the accompanying drawings.

[0050] Figure 1 This is a schematic diagram of an encoding / decoding system provided in this application. The encoding / decoding system 100 includes a source device 110 and a destination device 120. The source device 110 is used to compress and encode video or images to obtain a bitstream, and transmits the bitstream to the destination device 120. The destination device 120 decodes the bitstream, reconstructs the video or image, and displays the reconstructed image.

[0051] Specifically, the source device 110 includes an image acquisition unit 111, a preprocessor 112, an encoder 113, and a communication interface 114.

[0052] Image acquisition device 111 is used to acquire raw images. Image acquisition device 111 may include or be any type of image capture device for, for example, capturing real-world images, and / or any type of image or commentary (for screen content encoding, some text on the screen is also considered as an image to be encoded or part of an image) generation device, such as a computer graphics processor for generating computer-animated images, or any type of device for acquiring and / or providing real-world images, computer-animated images (e.g., screen content, virtual reality (VR) images), and / or any combination thereof (e.g., augmented reality (AR) images). Image acquisition device 111 may be a camera for capturing images or a memory for storing images. Image acquisition device 111 may also include any type of (internal or external) interface for storing previously captured or generated images and / or acquiring or receiving images. When the image acquisition device 111 is a camera, it may be, for example, a local camera or an integrated camera integrated into the source device; when the image acquisition device 111 is a memory, it may be a local memory or an integrated memory integrated into the source device. When the image acquisition device 111 includes an interface, the interface may be, for example, an external interface for receiving images from an external video source, such as an external image capture device, like a camera, external memory, or an external image generation device, such as an external computer graphics processor, computer, or server. The interface may be any type of interface according to any proprietary or standardized interface protocol, such as a wired or wireless interface, or an optical interface.

[0053] An image can be viewed as a two-dimensional array or matrix of pixels (picture elements). Pixels in an array are also called sample points. The number of sample points in the array or image along the horizontal and vertical directions (or axes) defines the image's size and / or resolution. To represent color, three color components are typically used; that is, an image can be represented as or contain three sample arrays. For example, in RBG format or color space, an image includes corresponding red, green, and blue sample arrays. However, in video coding, each pixel is typically represented in a luma / chroma format or color space. For example, for a YUV format image, this includes a luma component indicated by Y (sometimes also indicated by L) and two chroma components indicated by U and V. The luma component Y represents the brightness or grayscale level intensity (e.g., both are the same in a grayscale image), while the two chroma components U and V represent chroma or color information components. Accordingly, a YUV format image includes a luma sample array of luma sample values ​​(Y) and two chroma sample arrays of chroma values ​​(U and V). An RGB format image can be converted or transformed to YUV format, and vice versa; this process is also called color transformation or conversion. If the image is black and white, it may only include a luminance sampling array. In this application, the image transmitted from the image acquisition unit 111 to the encoder 113 can also be referred to as raw image data.

[0054] The preprocessor 112 receives the raw image acquired by the image acquisition unit 111 and preprocesses the raw image to obtain a preprocessed image. For example, the preprocessing performed by the preprocessor 112 includes retouching, color format conversion (e.g., from RGB format to YUV format), color adjustment, or noise reduction.

[0055] Encoder 113 is used to receive the preprocessed image generated by preprocessor 112 and compress and encode the preprocessed image to obtain a bitstream. For example, encoder 113 may include a code control unit 1131 and an encoding unit 1132. Code control unit 1131 is used to determine the quantization parameters used to encode each encoding unit in the current frame, so that encoding unit 1132 can predict, quantize, and encode the preprocessed image according to the quantization parameters to obtain a bitstream. Encoder 113 can determine the target number of bits based on the image content of the encoding unit and the number of bits in the bitstream buffer, and encode the encoding unit according to the quantization parameters determined by the target number of bits.

[0056] The communication interface 114 is used to receive the bit stream generated by the encoder 113 and send the bit stream to the destination device 120 through the communication channel 130 so that the destination device 120 can reconstruct the original image based on the bit stream.

[0057] The target device 120 includes a display 121, a post-processor 122, a decoder 123, and a communication interface 124.

[0058] Communication interface 124 is used to receive the bit stream sent by communication interface 114 and transmit the bit stream to decoder 123 so that decoder 123 can reconstruct the original image based on the bit stream.

[0059] Communication interfaces 114 and 124 can be used to send or receive data related to the original image through a direct communication link between the source device 110 and the destination device 120, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0060] Both communication interface 114 and communication interface 124 can be configured as follows: Figure 1 The arrow pointing from the source device 110 to the corresponding communication channel 130 of the destination device 120 indicates a one-way or two-way communication interface, which can be used to send and receive messages, establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission such as encoded bit stream transmission, etc.

[0061] Decoder 123 is used to decode the bitstream and reconstruct the original image. For example, decoder 123 performs entropy decoding, dequantization, and predictive reconstruction on the bitstream to obtain the reconstructed image. Decoder 123 may include a code control unit 1231 and a decoding unit 1232. Code control unit 1231 is used to determine the quantization parameters used for decoding each coding unit in the current frame, so that decoding unit 1232 can decode, dequantize, and predictively reconstruct the bitstream according to the quantization parameters to obtain the reconstructed image. Decoder 123 can determine the target number of bits for the coding unit based on the image content of the coding unit and the number of bits in the bitstream buffer, and decode the bitstream of the coding unit based on the quantization parameters determined by the target number of bits.

[0062] The post-processor 122 is used to receive the reconstructed image generated by the decoder 123 and perform post-processing on the reconstructed image. For example, the post-processing performed by the post-processor 122 includes color format conversion (e.g., from YUV format to RGB format), color adjustment, retouching or resampling, or any other processing.

[0063] Display 121 is used to display the reconstructed image. Display 121 can be or can include any class of display devices for presenting the reconstructed image, such as integrated or external displays or monitors. For example, the display can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other class of display.

[0064] Both encoder 113 and decoder 123 can be implemented as any of a variety of suitable circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof. If the technology is implemented in part in software, the device can store software instructions in a suitable non-transitory computer-readable storage medium, and one or more processors can be used to execute the instructions in hardware to perform the technology of this disclosure. Any of the foregoing (including hardware, software, combinations of hardware and software, etc.) can be considered as one or more processors.

[0065] The image acquisition unit 111 and the encoder 113 can be integrated into a single physical device or located on different physical devices; this is not limited. For example, such as... Figure 1The source device 110 shown includes an image acquisition unit 111 and an encoder 113, indicating that the image acquisition unit 111 and the encoder 113 are integrated into a single physical device. Therefore, the source device 110 can also be referred to as an acquisition device. The source device 110 can be, for example, a mobile phone, tablet computer, computer, laptop computer, camera, wearable device, in-vehicle device, terminal device, virtual reality (VR) device, augmented reality (AR) device, mixed reality (MR) device, extended reality (XR) device, or other image acquisition device. If the source device 110 does not include the image acquisition unit 111, it means that the image acquisition unit 111 and the encoder 113 are two different physical devices, and the source device 110 can acquire raw images from other devices (such as image acquisition devices or image storage devices).

[0066] Furthermore, the display 121 and the decoder 123 can be integrated into a single physical device or located on different physical devices; there is no limitation on this. For example, such as... Figure 1 The destination device 120 shown includes a display 121 and a decoder 123, indicating that the display 121 and decoder 123 are integrated into a single physical device. Therefore, the destination device 120 can also be referred to as a playback device. The destination device 120 has the function of decoding and displaying the reconstructed image. The destination device 120 can be, for example, a monitor, television, digital media player, video game console, in-vehicle computer, or other image display device. If the destination device 120 does not include the display 121, it means that the display 121 and decoder 123 are two different physical devices. After the destination device 120 decodes the bitstream and reconstructs the original image, it transmits the reconstructed image to other display devices (such as a television or digital media player) for display.

[0067] also, Figure 1 It is shown that the source device 110 and the destination device 120 can be integrated into one physical device or set on different physical devices, without limitation.

[0068] For example, such as Figure 2 As shown in (a), the source device 110 can be a camera, and the destination device 120 can be a display of various possible forms. The source device 110 can capture video of the first scene, transmit multiple frames of original images from the video to an encoding / decoding device, the encoding / decoding device performs encoding / decoding processing on the original images to obtain the reconstructed image, and the destination device 120 displays the reconstructed image and plays the video.

[0069] For example, such as Figure 2As shown in (b), if the source device 110 and the destination device 120 are integrated into a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, or an extended reality (XR) device, then the VR / AR / MR / XR device has the functions of acquiring original images, displaying reconstructed images, and encoding / decoding. The source device 110 can acquire images of the real-world scene in which the user is located, and the destination device 120 can display reconstructed images of the real-world scene in a virtual environment.

[0070] In these embodiments, the source device 110 or its corresponding functions and the destination device 120 or its corresponding functions may be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof. As described, Figure 1 The presence and division of different units or functions in the source device 110 and / or destination device 120 shown may vary depending on the actual device and application, which is obvious to those skilled in the art.

[0071] The structure of the above-described encoding / decoding system is only illustrative. In some possible implementations, the encoding / decoding system may also include other devices, such as end-side devices or cloud-side devices. After the source device 110 acquires the original image, it preprocesses the original image to obtain a preprocessed image; and then transmits the preprocessed image to the end-side device or cloud-side device, which performs encoding / decoding on the preprocessed image.

[0072] The image encoding and decoding method provided in this application is applied to both the encoding and decoding ends. Figure 3 The structure of the encoder and decoder is described in detail. For example... Figure 3 As shown, the encoder 300 includes a prediction unit 310, a code control unit 320, a quantization unit 330, an encoding unit 340, and a block partitioning unit 350.

[0073] Block partitioning unit 350 is used to divide the original image into multiple coding units.

[0074] The code control unit 320 is used to determine the target number of bits of the encoding unit based on the image content of the encoding unit that needs to be encoded and the number of bits of data in the bit stream buffer, as output by the block partitioning unit 350, and to determine the quantization parameters based on the target number of bits.

[0075] In some embodiments, when the encoder 300 transmits the bit stream of the image frame to the decoder 400 after encoding an image frame, the data in the bit stream buffer includes the bit stream of the image frame, and the number of bits in the data in the bit stream buffer includes the number of bits in the bit stream of the image frame.

[0076] In other embodiments, when encoder 300 transmits the bitstream of the encoded units to decoder 400 after encoding the encoded units in a frame image, the data in the bitstream buffer includes the bitstream of one or more encoded units, and the number of bits in the data in the bitstream buffer includes the number of bits of the bitstream of one or more encoded units. Understandably, the bitstream of one or more encoded units can be the bitstream of encoded units already encoded by encoder 300 minus the bitstream of encoded units transmitted by encoder 300 to decoder 400.

[0077] Prediction unit 310 performs intra-frame prediction on the coding units output by block partitioning unit 350 to obtain the predicted bit count, and outputs the residual between the original bit count and the predicted bit count of the coding unit. For example, an explanation of intra-frame prediction can be found in HEVC's intra-frame prediction. Intra-frame prediction is a common method for removing spatial redundancy information in the original image. It uses reconstructed pixels from adjacent coding blocks as reference values ​​to predict the current coding unit. This is equivalent to the coding unit in the original image having a correlation with its surrounding coding blocks. The pixel value of the current coding unit can be estimated using the reconstructed coding units around it. The estimated pixel value is the predicted value. The residual between the predicted value and the original value of the current coding unit is quantized and entropy encoded. The encoded data is often transmitted as the prediction residual. The decoding end performs the same prediction process to obtain the predicted value of the current coding unit, and then adds the predicted value to the solved residual to obtain the reconstructed value of the current coding unit.

[0078] The quantization unit 330 is used to quantize the residual output by the prediction unit 310 according to the quantization parameters output by the code control unit 320 to obtain the quantized residual.

[0079] The encoding unit 340 is used to encode the quantized residual output by the quantization unit 330 to obtain the bit stream of the encoding unit. For example, entropy encoding is performed on the quantized residual output by the quantization unit 330.

[0080] Combination Figure 3 The structure of the decoder is described in detail. For example... Figure 3 As shown, the decoder 400 includes a decoding unit 410, a code control unit 420, an inverse quantization unit 430, and a prediction and reconstruction unit 440.

[0081] The decoding unit 410 is used to decode the bit stream of the encoding unit to obtain the quantized residual and image content.

[0082] The code control unit 420 determines the target number of bits of the encoding unit based on the number of bits of data in the image content bitstream buffer of the encoding unit that needs to be decoded, as output by the decoding unit 410, and determines the quantization parameters based on the target number of bits.

[0083] In some embodiments, when the encoder 300 transmits the bit stream of the image frame to the decoder 400 after encoding an image frame, the data in the bit stream buffer of the decoder 400 includes the decoded data of the image frame, and the number of bits in the data in the bit stream buffer includes the number of bits of the decoded data of the image frame.

[0084] In other embodiments, when the encoder 300 transmits the bit stream of the encoding unit to the decoder 400 after encoding the encoding unit in a frame image, the data in the bit stream buffer of the decoder 400 includes the decoded data of one or more encoding units, and the number of bits in the data in the bit stream buffer includes the number of bits of the decoded data of one or more encoding units.

[0085] The dequantization unit 430 is used to dequantize the quantized residual output by the decoding unit 410 according to the quantization parameters output by the code control unit 420 to obtain the residual.

[0086] The prediction and reconstruction unit 440 is used to predict and reconstruct the image based on the residual output by the inverse quantization unit 430 so that the reconstructed image can be displayed on the display.

[0087] To address the challenge of determining the quantization parameters used for image encoding and decoding to ensure the quality of the reconstructed image, this application provides an image encoding and decoding method that considers both the image content of the encoding unit and the number of bits in the bitstream buffer by dynamically setting a target bit count. At the encoding end, the encoding unit is encoded using the quantization parameters determined by the target bit count. This reduces the number of encoded bits while ensuring the quality of the reconstructed image.

[0088] Next, the image encoding and decoding process will be explained with reference to the accompanying drawings. Figure 4 This is a flowchart illustrating an image encoding / decoding method provided in this application. Figure 1 The image encoding and decoding process performed by the source device 110 and the destination device 120 will be used as an example for illustration. Figure 4 As shown, the method includes the following steps.

[0089] Step 410: The source device 110 obtains the coding unit to be encoded in the current frame.

[0090] As described in the above embodiments, if the source device 110 carries an image acquisition device 111, the source device 110 can acquire raw images through the image acquisition device 111. Optionally, the source device 110 can also receive raw images acquired by other devices; or obtain raw images from the memory or other memory in the source device 110. The raw images may include at least one of real-world images acquired in real time, images stored in the device, and images synthesized from multiple images. This embodiment does not limit the method of acquiring raw images or the type of raw images.

[0091] The current frame refers to the frame or original image that is being encoded and decoded at the current moment. The preceding frame refers to the frame or original image that was encoded and decoded at a time prior to the current moment. The preceding frame can be a frame from the moment before the current moment or several moments before.

[0092] The source device 110 can divide the current frame into multiple coding units and encode the multiple coding units.

[0093] Step 420: The source device 110 determines the target number of bits based on the image content of the encoding unit and the number of bits of data in the bit stream buffer, and determines the quantization parameters based on the target number of bits.

[0094] The image content of an encoding unit indicates the complexity of different pixel regions within that unit. This includes, for example, the complexity of the pixel region's color, texture, and shape. In some embodiments, the encoder 113 divides the encoding unit into several sub-blocks. For each sub-block, the difference between adjacent pixel values ​​is calculated progressively in both horizontal and vertical directions. The absolute values ​​of these differences are summed to obtain the complexity level corresponding to the sub-block. This complexity level is then compared to a threshold to obtain the complexity level of the sub-block. After performing rule-based operations on the complexity levels of each sub-block, the complexity level k of the encoding unit is obtained. The complexity levels of different encoding units in the current frame can be different or the same.

[0095] The number of bits in the bitstream buffer indicates the number of bits in the bitstream of the encoded units in the current frame stored in the bitstream buffer. Before encoding, a physical buffer for storing the bitstream can be pre-configured in memory. The bitstream buffer can be obtained based on the number of bits in the bitstream of the encoded units stored in the physical buffer.

[0096] Source device 110 determines the target number of bits based on the complexity level indicated by the image content of the encoding unit and the number of bits of data in the bitstream buffer, and determines the quantization parameters based on the target number of bits. For a detailed explanation of determining the quantization parameters, please refer to the descriptions in steps 510 to 560 below.

[0097] Step 430: The source device 110 encodes the encoding unit according to the quantization parameters to obtain the bit stream of the encoding unit.

[0098] Source device 110 can perform encoding operations such as transformation, quantization, and entropy encoding on the encoding units to generate a bit stream, thereby achieving the purpose of data compression of the encoding units to be encoded. The number of bits in the bit stream of the encoding unit can be less than or greater than the target number of bits. For specific methods of generating the bit stream, please refer to the prior art and the description of the encoding unit 330 in the above embodiments.

[0099] Step 440: Source device 110 sends a bit stream to destination device 120.

[0100] Source device 110 can send the video bitstream to destination device 120 after all video encoding is completed. Alternatively, source device 110 can encode the original image in real time, frame by frame, and send the bitstream of each frame after encoding. Alternatively, source device 110 can encode the encoding units of the original image, and send the bitstream of each encoding unit after encoding. Specific methods for sending the bitstream can be found in existing technologies and the descriptions of communication interfaces 114 and 124 in the above embodiments.

[0101] Step 450: The destination device 120 acquires the bit stream of the encoding unit to be decoded in the image bit stream.

[0102] Step 460: The target device 120 determines the target number of bits based on the image content of the encoding unit and the number of bits of data in the bit stream buffer, and determines the quantization parameters based on the target number of bits of the encoding unit.

[0103] After receiving the bitstream of the encoding unit, the destination device 120 decodes the bitstream to obtain the image content and encoded data of the encoding unit. Then, it determines the quantization parameters based on the number of bits in the bitstream buffer and the image content of the encoding unit. The destination device 120 determines the target number of bits based on the complexity level indicated by the image content of the encoding unit and the number of bits in the bitstream buffer, and then determines the quantization parameters based on the target number of bits. For a detailed explanation of determining the quantization parameters, refer to steps 510 to 560 below.

[0104] Step 470: The destination device 120 decodes the bit stream of the coding unit of the current frame according to the quantization parameters to obtain the reconstructed image.

[0105] The target device 120 decodes the encoded data of the encoding unit according to the quantization parameters determined by the target number of bits of the encoding unit, and obtains the reconstructed image.

[0106] The destination device 120 displays the reconstructed image. Alternatively, the destination device 120 transmits the reconstructed image to another display device, which then displays the reconstructed image.

[0107] Thus, in order to obtain better encoding and decoding performance and the quality of the reconstructed image, the quantization parameters are dynamically set to balance the number of bits of data in the bitstream buffer and the image content of the coding unit, thereby reducing the number of encoded bits after encoding the image while ensuring the quality of the reconstructed image.

[0108] Next, the process of determining the quantization parameters will be explained in detail with reference to the accompanying drawings. Figure 5This is a flowchart illustrating an image encoding method provided in this application. Figure 3 The process of determining quantization parameters using the encoder 300 is illustrated using this example. Figure 5 The method and process are Figure 4 The specific operational procedures included in steps 420 and 460 are described below. Figure 5 As shown, the method includes the following steps.

[0109] Step 510: Encoder 300 determines the number of lossless bits of the encoding unit based on the image content of the encoding unit.

[0110] The image content of the coding unit is used to indicate the complexity of different pixel regions within the coding unit. The encoder 300 can determine the complexity level of the coding unit based on its image content, for example, as explained in step 420. The lossless bit count of the coding unit is used to indicate the expected bit count after lossless encoding of the coding unit.

[0111] In some embodiments, the lossless bit count can be a default value configured empirically. In other embodiments, the encoder 300 sets the expected bit count of the unencoded encoding unit after lossless encoding based on the bit count of the encoded encoding unit. The encoder 300 can determine the lossless bit count of the encoding unit by looking up a table based on the identifier and complexity level of the encoding unit. Assume B LL Indicates the number of lossless bits. B LL =Record BLL [T][k], where T represents the identifier of the encoding unit and k represents the complexity level of the encoding unit.

[0112] Step 520: Encoder 300 determines the number of lossy bits in the encoding unit based on the number of bits of data in the bit stream buffer.

[0113] The lossy bit count of a coding unit is used to indicate the expected number of bits after lossy encoding of the coding unit without referring to its content.

[0114] The encoder 300 determines the adjustment value of the lossy bit count based on the number of bits of the encoded units already encoded in the bitstream buffer. Furthermore, the encoder 300 determines the lossy bit count based on the initial value and the adjustment value of the lossy bit count. The lossy bit count satisfies the following formula (1).

[0115] Bpp = Bpp INI +Bpp ADJ Formula (1)

[0116] Where Bpp represents the number of lossy bits, Bpp INI The initial value for the lossy number of bits, Bpp ADJThis represents the adjustment value for the number of lossy bits. The initial value for the number of lossy bits is determined based on the number of bits in the coding unit and the compression ratio. The compression ratio is determined according to the requirements of the actual application scenario.

[0117] The adjustment value for the number of lossy bits and (RcBuf) END -RcBuf T Proportional to RcBuf END This indicates the expected number of bits in the bitstream buffer at the end of the current frame encoding or decoding. RcBuf T This represents the number of bits of the encoded units already encoded in the bitstream buffer. If RcBuf END -RcBuf T If the difference is greater than 0, it indicates that the number of bits of the encoded units in the bitstream buffer has not exceeded the expected number of bits in the bitstream buffer at the end of the current frame encoding or decoding, and more target bits can be allocated to the unencoded units; if RcBuf END -RcBuf T If the difference is less than 0, it indicates that the number of bits of the encoded units in the bitstream buffer exceeds the expected number of bits in the bitstream buffer at the end of the current frame encoding or decoding, and a smaller target number of bits can be allocated to the unencoded units; if RcBuf END -RcBuf T A difference of 0 indicates that the number of bits of the encoded units in the bitstream buffer is equal to the expected number of bits in the bitstream buffer at the end of the current frame encoding or decoding, which allows for the allocation of fewer target bits to unencoded units.

[0118] The number of bits of encoded units in the bitstream buffer is linearly obtained from the number of bits of encoded units in the physical buffer. For example, RcBuf T =PhyBuf T +X0, PhyBuf T This represents the number of bits of the encoded units already encoded in the physical buffer. The physical buffer refers to the storage space in memory used to store the bit stream of encoded units. The storage capacity of the physical buffer can be the number of bits of the bit stream of one or more encoded units. X0 represents a predefined parameter.

[0119] RcBuf corresponding to different coding units in the current frame END The same. The RcBuf corresponding to the coding unit in different frames. END They can be the same or different.

[0120] Step 530: Encoder 300 determines the amount of information based on the number of lost bits and the average number of lossless bits.

[0121] Information content represents the proportion of information of the current coding unit in the current frame, that is, the complexity of the content expressed by the coding unit in the current frame. For example, information content is the ratio of lossy bits to average lossless bits. Average lossless bits are used to indicate the expected number of bits after lossless encoding of the current frame. Average lossless bits can be the average expected number of bits after lossless encoding of each coding unit in the current frame. For example, the current frame includes coding unit 1 and coding unit 2. The expected number of bits after lossless encoding of coding unit 1 is 10 bits, and the expected number of bits after lossless encoding of coding unit 2 is 20 bits. The average lossless bit of the current frame is 15 bits. Information content represents the complexity level of the coding unit. Information content satisfies the following formula (2).

[0122] R = (Bpp*InvTab[X3*B AVG -bitsOffset-1]+X4)*X5 formula (2)

[0123] Where R represents the information content, Bpp represents the number of lossy bits, and B AVG This represents the average number of lossless bits. `bitsOffset` represents the offset, where `bitsOffset = BitsOffset - X1 * Bpp + X2`. `BitsOffset` represents the initial value of the offset, which is related to the image bit depth. `X1`, `X2`, `X3`, `X4`, and `X5` represent predefined parameters.

[0124] Step 540: Encoder 300 determines the target number of bits based on the lossless bit count and the information content of the encoding unit.

[0125] The target number of bits is used to indicate the expected number of bits after lossy encoding of the coding unit when referencing the content of the coding unit, that is, the expected number of bits after quantization encoding of the coding unit by the encoder 300 when referencing the content of the coding unit.

[0126] The larger the number of lossy bits, the larger the target number of bits; the smaller the number of lossy bits, the smaller the target number of bits. The target number of bits satisfies the following formula (3).

[0127] B TGT =(R*(B LL -bitsOffset)+X6)*X7 formula (3)

[0128] Among them, B TGT R represents the target number of bits, R represents the information content, and B represents the target number of bits. LL This represents the number of lossless bits, bitsOffset represents the offset, and X6 and X7 are predefined parameters.

[0129] Step 550: Encoder 300 clamps the target number of bits based on at least one of the lossy number of bits, the lossless number of bits, and the fullness of the bit stream buffer, to obtain the clamping value of the target number of bits.

[0130] Encoder 300 is based on the number of lossy bits Bpp and the number of lossless bits B. LL The minimum value B of the target number of bits is determined by the full size F of the bitstream buffer. MIN The maximum value B of the target number of bits MAX Furthermore, based on the minimum value B of the target number of bits... MIN The maximum value B of the target number of bits MAX For the target number of bits B TGT Perform clamping to obtain the clamping value B' for the target number of bits. TGT The clamping value for the target number of bits satisfies the following formula (4).

[0131] B' TGT =MIN(MAX(B MIN B TGT ),B MAX ) Formula (4)

[0132] For example, if B MIN >B TGT B MIN MAX The target number of bits is B. MIN For example, if B MIN >B TGT B MIN >B MAX The target number of bits is B. MAX For example, if B MIN TGT B TGT MAX The target number of bits is B. TGT .

[0133] The full extent of the bitstream buffer is used to indicate the storage status of the bitstream of encoded units in the bitstream buffer. The full extent of the bitstream buffer satisfies the following formula (5).

[0134] F = RcBuf T / RcBuf MAX Formula (5)

[0135] Where F represents the full size of the bitstream buffer, RcBuf T RcBuf represents the number of bits of the encoded units already encoded in the bitstream buffer. MAX ​​​This indicates the maximum number of bits allowed in the bitstream buffer. To maintain the number of bits in the physical buffer, if the bitstream buffer is too full, the expected number of bits after lossy encoding of the coding unit when referencing the content of the coding unit is reduced, thereby decreasing the number of bits in the physical buffer.

[0136] Step 560: Encoder 300 determines the quantization parameters based on the clamping values ​​of the lossless bit count and the target bit count.

[0137] When referencing the content of the coding unit, the encoder 300 clamps the expected number of bits after quantization of the coding unit, and then determines the quantization parameters based on the clamping values ​​of the lossless number of bits and the target number of bits. The quantization parameters satisfy formula (6).

[0138] QP = (B LL - B' TGT +X8)*X9)*X 10 Formula (6)

[0139] Among them, B LL B' represents the number of lossless bits. TGT Clamping values ​​representing the target number of bits, X8, X9, and X 10 These are pre-agreed parameters.

[0140] Understandably, the encoder determines the expected value of the encoded unit after encoding based on the complexity of the current encoding unit relative to the entire frame image; that is, it is derived from the complexity level of the encoding unit and the average complexity level of the entire frame. Therefore, to obtain better encoding performance and the quality of the reconstructed image, the code control module allocates different expected bit counts to different encoding units within the image, thereby maximizing the utilization of the specified total number of encoded bits and achieving the best possible quality for the decompressed image.

[0141] Figure 6 A flowchart illustrating the rate control process during decoding is shown. Figure 6 and Figure 5 The difference lies in the fact that the quantization parameters of the bitrate control output are used in the inverse quantization process. For the process of determining the quantization parameters, please refer to the above. Figure 5 The explanation in the text.

[0142] The following section discusses the minimum value B based on the target number of bits. MIN The maximum value B of the target number of bits MAX The method for determining the clamping value for the target number of bits is illustrated with an example. For example... Figure 7 As shown, the method for determining the clamping value of the target number of bits includes the following steps.

[0143] Step 710: Calculate the first minimum value B MIN1 For example, the first minimum value BMIN1 =(Param) Bpp ×Bpp+K1)×(MIN(B LL B LLMAX )×K2+K3), where B LL Bpp represents the number of lossless bits, and Bpp represents the number of lossy bits. LLMAX Param represents the maximum number of lossless bits after lossless encoding of the coding unit at the current pixel depth. Bpp These represent agreed-upon parameters at the current pixel depth, with K1, K2, and K3 representing these agreed-upon parameters.

[0144] Step 720: Calculate the second minimum value B MIN2 For example, the second minimum value B MIN2 = (K4-K5×F)×Bpp+K6, where F represents the full size of the bitstream buffer, Bpp represents the number of lossy bits, and K4, K5 and K6 represent agreed-upon parameters.

[0145] Step 730: Calculate the third minimum value B MIN3 For example, the third minimum value B MIN3 The calculation process is as follows: 1) Calculate the lossless bit difference B relative B relative =Clip(0,320,B) LLMAX –B LL (Where, Clip indicates that B can be used). LLMAX –B LL The value of B is limited to the range of 0 to 320. LL B represents the number of lossless bits. LLMAX This represents the maximum number of lossless bits that the encoding unit can achieve after lossless encoding at the current pixel depth. 2) Calculate the third minimum value B. MIN3 B MIN3 =Bpp-(K7×(FF) limit )+K8)-Sr×B relative Among them, F limit The upper limit of the agreed-upon bitstream buffer capacity is represented by Bpp, the number of lossy bits is represented by F, the full capacity of the bitstream buffer is represented by Sr, the parameter related to the image chroma sampling rate is represented by K7 and K8, and the agreed-upon parameters are represented by K7 and K8.

[0146] Step 740: Calculate the minimum value B MIN The minimum number of target bits, B. MIN =MAX(MAX(B) MIN1 B MIN2 ),B MIN3 ), among which, B MIN1 B represents the first minimum value calculated in step 710. MIN2B represents the second minimum value calculated in step 720. MIN3 This represents the third minimum value calculated in step 730.

[0147] Step 750: Calculate the first maximum value B MAX1 The offset value. For example, the offset value of the first maximum value, bppOffset1 = K9 - (K10 × F + K11) - Sr × B relative Where F represents the full size of the bitstream buffer, and B... relative The difference in the number of lossless bits calculated in step 730 is represented by Sr, which represents a parameter related to the image chroma sampling rate, and K9, K10 and K11 represent agreed-upon parameters.

[0148] Step 760: Calculate the second maximum value B MAX2 The offset value. For example, the offset value of the second maximum value, bppOffset2 = MAX(K12-Bpp,K13)-B. relative Where K12 and K13 represent agreed-upon parameters.

[0149] Step 770: Calculate the third maximum value B MAX3 The offset value. For example, the offset value of the third maximum value, bppOffset3 = K14 * (F limit -F), where F represents the full size of the input buffer and K14 represents the agreed-upon parameters.

[0150] Step 780: Calculate the maximum value B MAX The maximum number of target bits, B. MAX The calculation process is as follows: 1) Calculate the offset of the maximum value of the target number of bits, bppOffset = MIN(MIN(bppOffset1, bppOffset2), bppOffset3). Where bppOffset1 represents the first maximum value B calculated in step 750. MAX1 The offset value, bppOffset2, represents the second maximum value B calculated in step 760. MAX2 The offset value, bppOffset3, represents the third maximum value B calculated in step 770. MAX3 2) Calculate the maximum value B of the target number of bits. MAX B MAX =MAX(Bpp+bppOffset,B MIN ), B MIN This represents the minimum value B calculated in step 740. MIN .

[0151] Step 790: Target number of bits B TGT Perform clamping. Calculate the minimum value B in step 740. MINThe maximum value B calculated in step 780 MAX The range for clamping the target number of bits is used to obtain the clamping value of the target number of bits. This range is then used to clamp the target number of bits B. TGT Perform clamping, B' TGT =MIN(MAX(B MIN B TGT ),B MAX ).

[0152] It is understood that, in order to achieve the functions in the above embodiments, the encoder and decoder include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and method steps of the various examples described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0153] The above text combines Figures 1 to 7 The image encoding and decoding method provided according to this embodiment is described in detail below. Figure 8 This describes the encoding / decoding apparatus provided according to this embodiment.

[0154] Figure 8 This is a schematic diagram illustrating the structure of a possible encoding / decoding device provided in this embodiment. These encoding / decoding devices can be used to implement the functions of the encoder and decoder in the above method embodiments, and therefore can also achieve the beneficial effects of the above method embodiments. In this embodiment, the encoding / decoding device can be as follows: Figure 3 The encoder 300 and decoder 400 shown can also be modules (such as chips) applied to computing devices.

[0155] like Figure 8 As shown, the encoding / decoding device 800 includes a communication module 810, a code control module 820, an encoding module 830, and a storage module 840. The encoding / decoding device 800 is used to implement the above-mentioned... Figure 3 The embodiments shown depict the functions of the encoder 300 and decoder 400.

[0156] When the encoding / decoding device 800 is used to implement the functions of the encoder 300, the specific functions of each module are as follows.

[0157] The communication module 810 is used to acquire the coding units to be encoded in the current frame. For example, the communication module 810 is used to perform... Figure 4 Step 410.

[0158] The code control module 820 is used to determine the target number of bits for the encoding unit based on the image content of the encoding unit and the number of bits of data in the bitstream buffer, and to determine quantization parameters based on the target number of bits for the encoding unit. For example, the code control module 820 is used to execute... Figure 4 Step 420.

[0159] Encoding module 830 is used to encode the encoding unit according to the quantization parameters to obtain the bit stream of the encoding unit. For example, encoding module 830 is used to perform... Figure 4 Step 430.

[0160] When the encoding / decoding device 800 is used to implement the function of the decoder 400, the specific functions of each module are as follows.

[0161] The communication module 810 is used to acquire the bit stream of the encoded unit to be decoded in the image bit stream. For example, the communication module 810 is used to perform... Figure 4 Step 450.

[0162] The code control module 820 is used to determine the target number of bits for the encoding unit based on the image content of the encoding unit and the number of bits of data in the bitstream buffer, and to determine quantization parameters based on the target number of bits for the encoding unit. For example, the code control module 820 is used to execute... Figure 4 Step 460.

[0163] Encoding module 830 is used to decode the bitstream of the encoding unit according to the quantization parameters to obtain the reconstructed image of the encoding unit. For example, encoding module 830 is used to perform... Figure 4 Step 470.

[0164] The storage module 840 is used to store the number of bits of data in the bit stream buffer so that the code control module 820 can determine the quantization parameters.

[0165] It should be understood that the encoding / decoding device 800 in this application embodiment can be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. It can also be implemented using software. Figure 4The method shown, and its various modules, can also be software modules, and the encoding / decoding device 800, and its various modules can also be software modules.

[0166] The encoding / decoding apparatus 800 according to the embodiments of this application can correspondingly execute the methods described in the embodiments of this application, and the above and other operations and / or functions of each unit in the encoding / decoding apparatus 800 are respectively for implementing Figure 4 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0167] Figure 9 This is a schematic diagram of an image processing system provided in this application. Taking a mobile phone as an example, the system includes a memory 910, a processor 920, a sensor component 930, a multimedia component 940, and an input / output interface 950. The following description, in conjunction with... Figure 9 This section provides a detailed introduction to the various components of a mobile phone or the chip system built into it.

[0168] The memory 910 can be used to store data, software programs, and modules; it mainly includes a program storage area and a data storage area. The program storage area can store software programs, including instructions formed by code, including but not limited to the operating system and applications required for at least one function, such as sound playback function, image playback function, etc.; the data storage area can store data created according to the use of the mobile phone, such as audio data, image data, phonebook, etc. In the embodiments of this application, the memory 910 can be used to store the number of bits of data in the bit stream buffer, etc. In some feasible embodiments, there can be one memory or multiple memories; the memory can include floppy disks, hard disks such as internal hard disks and external hard disks, magnetic disks, optical disks, magneto-optical disks such as CD_ROM, DCD_ROM, non-volatile storage devices such as RAM, ROM, PROM, EPROM, EEPROM, flash memory, or any other form of storage medium known in the art.

[0169] The processor 920 is the control center of the mobile phone, connecting various parts of the device through various interfaces and lines. It executes software programs and / or software modules stored in the memory 910, and calls data stored in the memory 910 to perform various functions and process data, thereby providing overall monitoring of the mobile phone. In this application embodiment, the processor 920 can be used to execute one or more steps in the method embodiments of this application. For example, the processor 920 can be used to execute one or more steps 420 to 470 in the following method embodiments. In some feasible embodiments, the processor 920 can be a single-processor architecture, a multi-processor architecture, a single-threaded processor, or a multi-threaded processor, etc.; in some feasible embodiments, the processor 920 can include at least one of a central processing unit, a general-purpose processor, a digital signal processor, a neural network processor, an image processing unit, an image signal processor, a microcontroller, or a microprocessor, etc. In addition, the processor 920 may further include other hardware circuits or accelerators, such as application-specific integrated circuits, field-programmable gate arrays, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor 920 can also be a combination of functions that perform computing, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc.

[0170] Sensor component 930 includes one or more sensors for providing various aspects of the mobile phone's status assessment. Sensor component 930 may include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications, i.e., as part of a camera or webcam. In this embodiment, sensor component 930 can be used to support image acquisition by the webcam in multimedia component 940. Furthermore, sensor component 930 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor. Sensor component 930 can detect the mobile phone's acceleration / deceleration, orientation, on / off state, relative positioning of components, or temperature changes of the mobile phone.

[0171] The multimedia component 940 provides a screen that serves as an output interface between the mobile phone and the user. This screen can be a touch panel, and when it is a touch panel, it can be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. Furthermore, the multimedia component 940 includes at least one camera, for example, a front-facing camera and / or a rear-facing camera. When the mobile phone is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or the rear-facing camera can sense external multimedia signals, which are used to form image frames. Each front-facing and rear-facing camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0172] Input / output interface 950 provides an interface between processor 920 and peripheral interface modules, such as keyboards, mice, or USB (Universal Serial Bus) devices. In one possible implementation, input / output interface 950 may have only one input / output interface or multiple input / output interfaces.

[0173] Although not shown, the mobile phone may also include audio components and communication components, such as a microphone for audio components and a wireless fidelity (WiFi) module and a Bluetooth module for communication components, which will not be described in detail in the embodiments of this application.

[0174] The image processing system described above can be a general-purpose device or a dedicated device. For example, the image processing system can be an edge device (e.g., a box carrying a chip with processing capabilities). Alternatively, the image processing system can also be a server or other device with computing capabilities.

[0175] It should be understood that the image processing system according to this embodiment may correspond to the encoding / decoding device 800 in this embodiment, and may correspond to the execution of the system according to this embodiment. Figure 3 The corresponding subject in any of the methods, and the above and other operations and / or functions of each module in the encoding / decoding device 800 are respectively for implementing Figure 3 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0176] The method steps in this embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a computing device. Of course, the processor and storage medium can also exist as discrete components in the computing device.

[0177] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD). The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image decoding method, characterized in that, include: Obtain the bitstream of the encoded unit to be decoded in the image bitstream; The lossless bit count of the encoding unit is determined based on the complexity level of the encoding unit. The lossless bit count of the encoding unit is used to indicate the expected bit count after lossless encoding of the encoding unit with reference to the complexity level of the encoding unit. The amount of information of the coding unit in the current frame is determined based on the number of bits of data in the bitstream buffer. The amount of information is used to indicate the proportion of the content expressed by the coding unit in the current frame. The target number of bits for the encoding unit is determined based on the lossless bit count and the information content of the encoding unit. The quantization parameters are determined based on the target number of bits of the coding unit, and the quantization parameters are used to decode the bit stream of the coding unit. The bitstream buffer is used to store the number of encoded bits after decoding one or more encoding units, and the target number of the encoding unit is used to indicate the expected number of bits after lossy encoding of the encoding unit with reference to the complexity level of the encoding unit.

2. The method according to claim 1, characterized in that, Determining the amount of information of the coding unit in the current frame based on the number of bits of data in the bitstream buffer includes: The lossy bit count of the encoding unit is determined based on the number of bits of data in the bitstream buffer. The lossy bit count of the encoding unit is used to indicate the expected number of bits after lossy encoding of the encoding unit without referring to the complexity level of the encoding unit. The information content is determined based on the lossy number of bits and the average lossless number of bits of the coding unit, wherein the average lossless number of bits is used to indicate the average expected number of bits after lossless coding of each coding unit in the current frame.

3. The method according to claim 2, characterized in that, Determining quantization parameters based on the target number of bits of the coding unit includes: The target number of bits of the encoding unit is clamped according to at least one of the lossless number of bits of the encoding unit, the lossy number of bits of the encoding unit, and the full value of the bit stream buffer, to obtain the clamped target number of bits. The full value of the bit stream buffer is used to indicate the ratio of the number of bits of data in the bit stream buffer to the storage capacity of the bit stream buffer. The quantization parameters are determined based on the lossless number of bits in the encoding unit and the target number of bits after clamping.

4. The method according to claim 3, characterized in that, The complexity level of the encoding unit includes at least one of luminance complexity level and chrominance complexity level.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: The bitstream of the coding unit is decoded according to the quantization parameters to obtain the reconstructed image of the coding unit.

6. An image encoding method, characterized in that, include: Obtain the encoding unit to be encoded in the current frame; The lossless bit count of the encoding unit is determined based on the complexity level of the encoding unit, and the lossless bit count of the encoding unit is used to indicate the expected bit count after lossless encoding of the encoding unit; The amount of information of the coding unit in the current frame is determined based on the number of bits of data in the bitstream buffer. The amount of information is used to indicate the proportion of the content expressed by the coding unit in the current frame. The target number of bits for the encoding unit is determined based on the lossless bit count and the information content of the encoding unit. The quantization parameters are determined based on the target number of bits of the coding unit, and the quantization parameters are used to encode the coding unit; The bitstream buffer is used to store the bitstream or part of the bitstream of one or more coding units, and the target number of bits of the coding unit is used to indicate the expected number of bits after lossy coding of the coding unit with reference to the complexity level of the coding unit.

7. The method according to claim 6, characterized in that, Determining the amount of information of the coding unit in the current frame based on the number of bits of data in the bitstream buffer includes: The lossy bit count of the encoding unit is determined based on the number of bits of data in the bitstream buffer. The lossy bit count of the encoding unit is used to indicate the expected number of bits after lossy encoding of the encoding unit without referring to the complexity level of the encoding unit. The information content is determined based on the lossy number of bits and the average lossless number of bits of the coding unit, wherein the average lossless number of bits is used to indicate the average expected number of bits after lossless coding of each coding unit in the current frame.

8. The method according to claim 7, characterized in that, Determining quantization parameters based on the target number of bits of the coding unit includes: The target number of bits of the encoding unit is clamped according to at least one of the lossless number of bits of the encoding unit, the lossy number of bits of the encoding unit, and the full value of the bit stream buffer, to obtain the clamped target number of bits. The full value of the bit stream buffer is used to indicate the ratio of the number of bits of data in the bit stream buffer to the storage capacity of the bit stream buffer. The quantization parameters are determined based on the lossless number of bits in the encoding unit and the target number of bits after clamping.

9. The method according to claim 8, characterized in that, The complexity level of the encoding unit includes at least one of luminance complexity level and chrominance complexity level.

10. The method according to any one of claims 6-9, characterized in that, The method further includes: The encoding unit is encoded according to the quantization parameters to obtain the bit stream of the encoding unit.

11. A codec device, characterized in that, include: The communication module is used to acquire the bit stream of the encoded units to be decoded in the image bit stream; The code control module is used to determine the number of lossless bits of the encoding unit according to the complexity level of the encoding unit. The number of lossless bits of the encoding unit is used to indicate the expected number of bits after lossless encoding of the encoding unit with reference to the complexity level of the encoding unit. The code control module is further configured to determine the amount of information of the coding unit in the current frame based on the number of bits of data in the bit stream buffer. The amount of information is used to indicate the proportion of the content expressed by the coding unit in the content expressed in the current frame. The code control module is also used to determine the target number of bits of the encoding unit based on the lossless number of bits of the encoding unit and the information content of the encoding unit; The code control module is further configured to determine quantization parameters based on the target number of bits of the encoding unit, and the quantization parameters are used to decode the bit stream of the encoding unit; The bitstream buffer is used to store the number of encoded bits after decoding one or more encoding units, and the target number of the encoding unit is used to indicate the expected number of bits after lossy encoding of the encoding unit with reference to the complexity level of the encoding unit.

12. The apparatus according to claim 11, characterized in that, When the code control module determines the amount of information of the coding unit in the current frame based on the number of bits of data in the bit stream buffer, it is specifically used for: The lossy bit count of the encoding unit is determined based on the number of bits of data in the bitstream buffer. The lossy bit count of the encoding unit is used to indicate the expected number of bits after lossy encoding of the encoding unit without referring to the complexity level of the encoding unit. The information content is determined based on the lossy number of bits and the average lossless number of bits of the coding unit, wherein the average lossless number of bits is used to indicate the average expected number of bits after lossless coding of each coding unit in the current frame.

13. The apparatus according to claim 12, characterized in that, When the code control module determines the quantization parameters based on the target number of bits of the coding unit, it is specifically used for: The target number of bits of the encoding unit is clamped according to at least one of the lossless number of bits of the encoding unit, the lossy number of bits of the encoding unit, and the full value of the bit stream buffer, to obtain the clamped target number of bits. The full value of the bit stream buffer is used to indicate the ratio of the number of bits of data in the bit stream buffer to the storage capacity of the bit stream buffer. The quantization parameters are determined based on the lossless number of bits in the encoding unit and the target number of bits after clamping.

14. The apparatus according to claim 13, characterized in that, The complexity level of the encoding unit includes at least one of luminance complexity level and chrominance complexity level.

15. The apparatus according to any one of claims 11-14, characterized in that, The device also includes an encoding module; The encoding module is used to decode the bitstream of the encoding unit according to the quantization parameters to obtain the reconstructed image of the encoding unit.

16. A codec device, characterized in that, include: The communication module is used to obtain the encoding units to be encoded in the current frame; The code control module is used to determine the number of lossless bits of the encoding unit according to the complexity level of the encoding unit, and the number of lossless bits of the encoding unit is used to indicate the expected number of bits after lossless encoding of the encoding unit; The code control module is further configured to determine the amount of information of the coding unit in the current frame based on the number of bits of data in the bit stream buffer. The amount of information is used to indicate the proportion of the content expressed by the coding unit in the content expressed in the current frame. The code control module is also used to determine the target number of bits of the encoding unit based on the lossless number of bits of the encoding unit and the information content of the encoding unit; The code control module is further configured to determine quantization parameters based on the target number of bits of the coding unit, and the quantization parameters are used to encode the coding unit; The bitstream buffer is used to store the bitstream or part of the bitstream of one or more coding units, and the target number of bits of the coding unit is used to indicate the expected number of bits after lossy coding of the coding unit with reference to the complexity level of the coding unit.

17. The apparatus according to claim 16, characterized in that, When the code control module determines the amount of information of the coding unit in the current frame based on the number of bits of data in the bit stream buffer, it is specifically used for: The lossy bit count of the encoding unit is determined based on the number of bits of data in the bitstream buffer. The lossy bit count of the encoding unit is used to indicate the expected number of bits after lossy encoding of the encoding unit without referring to the complexity level of the encoding unit. The information content is determined based on the lossy number of bits and the average lossless number of bits of the coding unit, wherein the average lossless number of bits is used to indicate the average expected number of bits after lossless coding of each coding unit in the current frame.

18. The apparatus according to claim 17, characterized in that, When the code control module determines the quantization parameters based on the target number of bits of the coding unit, it is specifically used for: The target number of bits of the encoding unit is clamped according to at least one of the lossless number of bits of the encoding unit, the lossy number of bits of the encoding unit, and the full value of the bit stream buffer, to obtain the clamped target number of bits. The full value of the bit stream buffer is used to indicate the ratio of the number of bits of data in the bit stream buffer to the storage capacity of the bit stream buffer. The quantization parameters are determined based on the lossless number of bits in the encoding unit and the target number of bits after clamping.

19. The apparatus according to claim 18, characterized in that, The complexity level of the encoding unit includes at least one of luminance complexity level and chrominance complexity level.

20. The apparatus according to any one of claims 16-19, characterized in that, The device also includes an encoding module; The encoding module is used to encode the encoding unit according to the quantization parameters to obtain the bit stream of the encoding unit.

21. A decoder, characterized in that, The decoder includes at least one processor and a memory, wherein the memory is used to store a computer program such that when the computer program is executed by the at least one processor, it implements the method as described in any one of claims 1-5.

22. An encoder, characterized in that, The encoder includes at least one processor and a memory, wherein the memory is used to store a computer program such that when the computer program is executed by the at least one processor, it implements the method as described in any one of claims 6-10.

23. A codec system, characterized in that, The encoding / decoding system includes an encoder as described in claim 22 and a decoder as described in claim 21, wherein the encoder is used to perform the operation steps of the method according to any one of claims 6-10, and the decoder is used to perform the method according to any one of claims 1-5.

24. A chip, characterized in that, include: A processor and a power supply circuit; wherein the power supply circuit is used to supply power to the processor; The processor is used to perform the operation steps of the method according to any one of claims 6-10, or to perform the method according to any one of claims 1-5.

25. A computer-readable storage medium, characterized in that, include: Computer software instructions; when executed in a computing device, the computer software instructions cause the computing device to perform the operational steps of the method according to any one of claims 6-10, or to perform the method according to any one of claims 1-5.

26. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computing device to perform the operation steps of the method according to any one of claims 6-10, or to perform the method according to any one of claims 1-5.

Images