Coding and decoding method and device

Abstract

The invention provides a coding and decoding method and device, relates to the technical field of media, and can improve the quality of a video picture to a certain extent in a code rate fluctuation scene. The method comprises the following steps: analyzing a code stream, and obtaining variable resolution coding indication information which is used for indicating the resolution of a current coding image; determining a predicted value of a current coding unit in the current coding image according to the resolution of the current coding image and the resolution of a reference image of the current coding image; wherein under the condition that the resolution of the current coding image is smaller than the original resolution, the maximum size of the current coding unit is the first size.

Description

[0001] This application claims the priority of a Chinese patent application with the application number 202410107247.7 and the application title "A Coding and Decoding Method and Device" filed with the National Intellectual Property Administration on January 24, 2024, the entire content of which is incorporated herein by reference. Technical Field

[0002] This application relates to the field of media technology, and in particular, to a coding and decoding method and device. Background Art

[0003] With more and more wireless devices being applied to security work, wireless video transmission has increasingly become a topic of our concern. However, real-time video transmission will inevitably encounter unexpected factors, which cause the bitrate of video coding to suddenly decrease, thereby leading to a decline in the picture quality.

[0004] How to reduce the impact of bitrate fluctuations on video picture quality is an issue that needs to be considered in the coding and decoding process. Summary of the Invention

[0005] This application provides a coding and decoding method and device, which can improve the quality of video pictures to a certain extent in the scenario of bitrate fluctuations.

[0006] This application adopts the following technical solutions:

[0007] In a first aspect, this application provides a decoding method, including: parsing a bitstream to obtain variable-resolution coding indication information, where the variable-resolution coding indication information is used to indicate the resolution of the current coded image; then, determining the motion vector of the current coding unit in the current coded image; where, when the resolution of the current coded image is less than the original resolution, the maximum size of the current coding unit is a first size; scaling the motion vector of the current coding unit according to the resolution of the current coded image and the resolution of the reference image of the current coded image; and then determining the predicted value of the current coding unit based on the scaled motion vector.

[0008] In the coding method provided by this application, the image can be encoded with reduced resolution, and the resolution of the coded image is indicated in the bitstream. In this way, in the scenario of bitrate fluctuations, through variable-resolution coding, without increasing the quantization parameter QP, the quality of video pictures can be improved to a certain extent.

[0009] In a second aspect, the present application provides a decoding method, including: parsing a bitstream to obtain variable resolution coding indication information, where the variable resolution coding indication information is used to indicate the resolution of the current coded image; and determining a prediction value of a current coding unit in the current coded image according to the resolution of the current coded image and the resolution of a reference image of the current coded image; where, when the resolution of the current coded image is less than the original resolution, the maximum size of the current coding unit is a first size.

[0010] In the first and second aspects, the coding method provided by the present application can perform downsampling coding on an image and indicate the resolution of the coded image in the bitstream. Thus, in a scenario with bitrate fluctuations, through variable resolution coding, without increasing the quantization parameter QP, the quality of the video picture can be improved to a certain extent.

[0011] In a possible implementation, determining a prediction value of a current coding unit in the current coded image according to the resolution of the current coded image and the resolution of a reference image of the current coded image includes: determining a motion vector of the current coding unit; determining a scaling parameter according to the resolution of the current coded image and the resolution of a reference image of the current coded image; and determining the prediction value of the current coding unit according to the scaling parameter and the motion vector of the current coding unit. The scaling parameter includes a scaling flag (which can be denoted as scaleFlag) and a scaling offset flag (scaleShift), where the scaling flag is used to indicate whether to perform a scaling operation when coding the current image, and the scaling offset flag is used to indicate the number of bits of the scaling offset.

[0012] In a third aspect, the present application provides a coding method, including: coding a current image to generate a bitstream; the bitstream includes a current coded image and variable resolution coding indication information, where the variable resolution coding indication information is used to indicate the resolution of the current coded image; where, when the resolution of the current coded image is less than the original resolution, the maximum size of the current coding unit is a first size; and transmitting the bitstream.

[0013] In the decoding method provided by the present application, indication information indicating the resolution of the coded image can be parsed from the bitstream, and prediction of the coding unit can be performed based on the resolution indicated by the indication information and the resolution of the reference image to complete decoding. This method can, in a scenario with bitrate fluctuations, through variable resolution coding, without increasing the quantization parameter QP, improve the quality of the video picture to a certain extent.

[0014] In a possible implementation of the first aspect and the second aspect, the first dimension (the maximum dimension of the current coding unit) is 64×64. If both the width and height of the maximum coding unit are equal to 128, and the value of the variable resolution coding indication information PictureRprFlag for this image is 1 (the horizontal dimension and the vertical dimension of the current image are equal to 1 / 2 of the horizontal dimension of the sequence parameter set, and the vertical dimension of the current image is equal to 1 / 2 of the vertical dimension of the sequence parameter set), then the maximum coding unit should use quadtree partitioning.

[0015] In a possible implementation of the first aspect and the second aspect, when the value of the variable resolution coding indication information is the first value, the resolution of the current coded image is equal to 1 / 2 times the original resolution; when the value of the variable resolution coding indication information is the second value, the resolution of the current coded image is equal to the original resolution.

[0016] In a possible implementation of the first aspect and the second aspect, when the value of the variable resolution coding indication information is the first value, the resolution of the current coded image is equal to 1 / 4 times the original resolution; when the value of the variable resolution coding indication information is the second value, the resolution of the current coded image is equal to the original resolution.

[0017] In a possible implementation of the first aspect and the second aspect, the coded dimension of the current coded image is less than or equal to the sum of half of the original coded dimension of the current coded image and the dimension of the maximum coding unit, and greater than or equal to half of the original coded dimension of the current coded image.

[0018] In a fourth aspect, the present application provides a decoding device, which includes various modules for implementing the method described in one of the first aspect and its possible implementations. The decoding device has the function of implementing the behavior in the method example of any one of the above-mentioned first aspect, second aspect, and their possible implementations. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0019] In a fifth aspect, the present application provides an encoding device, which includes various modules for implementing the method described in one of the second aspect and its possible implementations. The encoding device has the function of implementing the behavior in the method example of any one of the above-mentioned third aspect and its possible implementations. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0020] Sixth aspect, the present application provides a decoding device, including at least one processor and a memory. The at least one processor executes a program or instructions stored in the memory, so that the decoding device implements the method described in any one of the above first aspect, second aspect or any possible implementation manners thereof.

[0021] Seventh aspect, the present application provides an encoding device, including at least one processor and a memory. The at least one processor executes a program or instructions stored in the memory, so that the encoding device implements the method described in any one of the above third aspect or any possible implementation manners thereof.

[0022] Eighth aspect, an embodiment of the present application further provides a computer-readable storage medium for storing a computer program, and the computer program includes a method for implementing any one of the above first aspect, second aspect, third aspect or any possible implementation manners thereof.

[0023] Ninth aspect, an embodiment of the present application further provides a computer program product including instructions, and when it runs on a computer, it enables the computer to implement the method described in any one of the above first aspect, second aspect, third aspect or any possible implementation manners thereof.

[0024] Tenth aspect, an embodiment of the present application further provides a chip, including: an input interface, an output interface, and at least one processor. Optionally, the chip further includes a memory. The at least one processor is configured to execute code in the memory, and when the at least one processor executes the code, the chip implements the method described in any one of the above first aspect, second aspect, third aspect or any possible implementation manners thereof.

[0025] Optionally, the above chip may also be an integrated circuit.

[0026] The decoding device, encoding device, computer storage medium, computer program product and chip provided by the present application are all used to execute the decoding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the decoding method provided above, and will not be elaborated here. Description of the Drawings

[0027] Figure 1a It is an exemplary block diagram of a decoding system provided by an embodiment of the present application;

[0028] Figure 1b It is an exemplary block diagram of a video decoding system provided by an embodiment of the present application;

[0029] Figure 2 It is an exemplary block diagram of a video encoder provided by an embodiment of the present application;

[0030] Figure 3An exemplary block diagram of a video decoder provided by an embodiment of the present application;

[0031] Figure 4 An exemplary schematic diagram of a candidate image block provided by an embodiment of the present application;

[0032] Figure 5 An exemplary block diagram of a video decoding device provided by an embodiment of the present application;

[0033] Figure 6 An exemplary block diagram of a device provided by an embodiment of the present application;

[0034] Figure 7 A schematic flowchart of an encoding method provided by an embodiment of the present application;

[0035] Figure 8 A schematic diagram of a process for encoding an image provided by an embodiment of the present application;

[0036] Figure 9 A schematic diagram of a process for motion compensation provided by an embodiment of the present application;

[0037] Figure 10 A schematic flowchart of a decoding method provided by an embodiment of the present application;

[0038] Figure 11 A schematic flowchart of another decoding method provided by an embodiment of the present application;

[0039] Figure 12 A schematic diagram of a process for decoding an image provided by an embodiment of the present application;

[0040] Figure 13 A schematic flowchart of a third decoding method provided by an embodiment of the present application;

[0041] Figure 14 A schematic diagram of an encoding device provided by an embodiment of the present application;

[0042] Figure 15 A schematic diagram of a decoding device provided by an embodiment of the present application;

[0043] Figure 16 A schematic diagram of the structure of a chip provided by an embodiment of the present application;

[0044] Figure 17 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application;

[0045] Figure 18 A schematic diagram of the structure of an image processing device provided by an embodiment of the present application. Detailed implementation manners

[0046] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the embodiments of the present application without creative efforts belong to the scope protected by the embodiments of the present application.

[0047] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone.

[0048] The terms "first" and "second" in the description of the embodiments of the present application and the accompanying drawings are used to distinguish different objects or different treatments of the same object, rather than to describe the specific order of the objects.

[0049] In addition, the terms "include" and "have" and any variations thereof mentioned in the description of the embodiments of the present application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes other unlisted steps or units, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.

[0050] It should be noted that in the description of the embodiments of the present application, words such as "exemplarily" or "for example" are used to give examples, illustrations, or explanations. Any embodiment or design solution described as "exemplarily" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplarily" or "for example" is intended to present relevant concepts in a specific manner.

[0051] Data encoding and decoding includes two parts: data encoding and data decoding. Data encoding is performed on the source side (or usually referred to as the encoder side), and generally includes processing (for example, compressing) the original data to reduce the amount of data required to represent the original data (thereby more efficiently storing and / or transmitting). Data decoding is performed on the destination side (or usually referred to as the decoder side), and generally includes performing inverse processing relative to the encoder side to reconstruct the original data. The "encoding and decoding" of the data involved in the embodiments of the present application should be understood as "encoding" or "decoding" of the data. The encoding part and the decoding part are also collectively referred to as encoding and decoding (encoding and decoding, CODEC).

[0052] In the case of lossless data encoding, the original data can be reconstructed, that is, the reconstructed original data has the same quality as the original data (assuming no transmission loss or other data loss during storage or transmission). In the case of lossy data encoding, further compression is performed through quantization, etc., to reduce the amount of data required to represent the original data, and the decoder side cannot fully reconstruct the original data, that is, the quality of the reconstructed original data is lower or worse than that of the original data.

[0053] The embodiments of the present application can be applied to video data and other data with compression / decompression requirements, etc. The following takes the encoding of video data (abbreviated as video encoding) as an example to illustrate the embodiments of the present application. Other types of data (such as image data, audio data, integer data, and other data with compression / decompression requirements) can refer to the following description, and the embodiments of the present application will not be elaborated herein. It should be noted that compared with video encoding, during the encoding process of data such as audio data and integer data, the data does not need to be divided into blocks, but can be directly encoded.

[0054] Video encoding generally refers to processing an image sequence that forms a video or video sequence. In the field of video encoding, the terms "picture", "frame", or "image" can be used as synonyms.

[0055] Several video encoding standards belong to "lossy hybrid video codecs" (that is, combining spatial and temporal prediction in the pixel domain with 2D transform coding for applying quantization in the transform domain). Each image in a video sequence is usually divided into a set of non-overlapping blocks, and encoding is usually performed at the block level. In other words, the encoder usually processes and encodes the video at the block (video block) level. For example, predictive blocks are generated through spatial (intra-frame) prediction and temporal (inter-frame) prediction; the predictive blocks are subtracted from the current block (the currently processed / block to be processed) to obtain a residual block; the residual block is transformed and quantized in the transform domain to reduce the amount of data to be transmitted (compressed), and the decoder side applies the inverse processing part relative to the encoder to the encoded or compressed block to reconstruct the current block for representation. In addition, the encoder needs to repeat the processing steps of the decoder so that the encoder and the decoder generate the same predictions (such as intra-frame prediction and inter-frame prediction) and / or reconstruct pixels for processing, that is, encoding subsequent blocks.

[0056] In the following embodiments of the decoding system 10, the encoder 20 and the decoder 30 are described according to Figures 1a to 3 for description.

[0057] Figure 1aAn exemplary block diagram of the decoding system 10 provided by an embodiment of the present application. For example, a video decoding system 10 (or simply referred to as the decoding system 10) that can utilize the technology of the embodiments of the present application. The video encoder 20 (or simply referred to as the encoder 20) and the video decoder 30 (or simply referred to as the decoder 30) in the video decoding system 10 represent devices and the like that can be used to execute various techniques according to the various examples described in the embodiments of the present application.

[0058] As Figure 1a shown, the decoding system 10 includes a source device 12, and the source device 12 is used to provide encoded image data 21 such as encoded images to a destination device 14 for decoding the encoded image data 21.

[0059] The source device 12 includes an encoder 20, and additionally, optionally, may include an image source 16, a pre-processor (or pre-processing unit) 18 such as an image pre-processor, and a communication interface (or communication unit) 22.

[0060] The image source 16 may include or can be any type of image capture device for capturing real-world images and the like, and / or any type of image generation device, such as a computer graphics processor for generating computer animated images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The above image source can be any type of memory or storage for storing any of the above images.

[0061] To distinguish the processing performed by the pre-processor (or pre-processing unit) 18, the image (or image data) 17 may also be referred to as the original image (or original image data) 17.

[0062] The pre-processor 18 is used to receive the original image data 17 and pre-process the original image data 17 to obtain pre-processed images (or pre-processed image data) 19. For example, the pre-processing performed by the pre-processor 18 may include trimming, color format conversion (e.g., from RGB to YCbCr), color correction, or denoising. It can be understood that the pre-processing unit 18 can be an optional component.

[0063] The video encoder (or encoder) 20 is used to receive the pre-processed image data 19 and provide the encoded image data 21 (which will be further described below according to Figure 2 etc.).

[0064] The communication interface 22 in the source device 12 can be used to: receive the encoded image data 21 and send the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device through the communication channel 13 for storage or direct reconstruction.

[0065] The destination device 14 includes a decoder 30, and additionally, optionally, may include a communication interface (or communication unit) 28, a post-processor (or post-processing unit) 32, and a display device 34.

[0066] The communication interface 28 in the destination device 14 is used to directly receive the encoded image data 21 (or any other processed version) from the source device 12 or from any other source device such as a storage device. For example, the storage device is an encoded image data storage device, and provide the encoded image data 21 to the decoder 30.

[0067] The communication interface 22 and the communication interface 28 can be used to send or receive the encoded image data (or encoded data) 21 through a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, etc., or through any type of network, such as a wired network, a wireless network, or any combination thereof, any type of private network and public network, or any combination of any type thereof.

[0068] For example, the communication interface 22 can be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or use any type of transmission encoding or processing to process the above-mentioned encoded image data for transmission on the communication link or communication network.

[0069] The communication interface 28 corresponds to the communication interface 22. For example, it can be used to receive the transmission data, and use any type of corresponding transmission decoding or processing and / or de-encapsulation to process the transmission data to obtain the encoded image data 21.

[0070] Both the communication interface 22 and the communication interface 28 can be configured as Figure 1a a unidirectional communication interface or a bidirectional communication interface as indicated by the arrow of the corresponding communication channel 13 pointing from the source device 12 to the destination device 14 in

[0071] and can be used to send and receive messages, etc., to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission such as the transmission of the encoded image data, etc. Figure 3 etc. will be further described below.

[0072] The post-processor 32 is used to perform post-processing on the decoded image, such as the decoded image data 31 (also referred to as the reconstructed image data), to obtain the post-processed image, such as the post-processed image data 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color grading, cropping, or resampling, or any other processing for generating the decoded image data 31 to be displayed on the display device 34, etc.

[0073] The display device 34 is used to receive the post-processed image data 33 to display an image to a user, viewer, etc. The display device 34 may be or include any type of display for representing the reconstructed image. For example, an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0074] The decoding system 10 further includes a training engine 25, which is used to train the encoder 20 (especially the entropy encoding unit 270 in the encoder 20) or the decoder 30 (especially the entropy decoding unit 304 in the decoder 30) to perform entropy encoding on the image block to be encoded according to the estimated probability distribution. For a detailed description of the training engine 25, please refer to the following method test examples.

[0075] Although Figure 1a the source device 12 and the destination device 14 are shown as separate devices, the device embodiments may also include both the source device 12 and the destination device 14 or the functions of both the source device 12 and the destination device 14 at the same time, that is, including both the source device 12 or the corresponding function and the destination device 14 or the corresponding function at the same time. In these embodiments, the source device 12 or the corresponding function and the destination device 14 or the corresponding function may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.

[0076] According to the description, Figure 1a the presence and (accurate) division of different units or functions in the shown source device 12 and / or destination device 14 may vary according to the actual device and application, which is obvious to those skilled in the art.

[0077] Please refer to Figure 1b , Figure 1bAn exemplary block diagram of the video decoding system 40 provided by an embodiment of the present application. The encoder 20 (such as the video encoder 20), or the decoder 30 (such as the video decoder 30), or both can be implemented by a processing circuit in the video decoding system 40 as shown in Figure 1b , for example, one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, video encoding dedicated processors, or any combination thereof. Please refer to Figure 2 and Figure 3 . Figure 2 An exemplary block diagram of the video encoder provided by an embodiment of the present application. Figure 3 An exemplary block diagram of the video decoder provided by an embodiment of the present application. The encoder 20 can be implemented by the processing circuit 46 to include various modules discussed with reference to Figure 2 the encoder 20 and / or any other encoder system or subsystem described herein. The decoder 30 can be implemented by the processing circuit 46 to include various modules discussed with reference to Figure 3 the decoder 30 and / or any other decoder system or subsystem described herein. The above-mentioned processing circuit 46 can be used to perform various operations discussed below. As shown in Figure 5 , if part of the technology is implemented in software, the device can store the instructions of the software in a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors, thereby implementing the technology of the embodiments of the present application. One of the video encoder 20 and the video decoder 30 can be integrated as part of a combined codec (encoder / decoder, CODEC) in a single device, as shown in Figure 1b .

[0078] The source device 12 and the destination device 14 may include any of a variety of devices, including any type of handheld device or fixed device, such as, for example, a laptop or notebook computer, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (e.g., a content service server or a content distribution server), a broadcast receiving device, a broadcast transmitting device, and a video capture device, etc., and may or may not use any type of operating system. The source device 12 and the destination device 14 may also be devices in a cloud computing scenario, such as virtual machines in a cloud computing scenario. In some cases, the source device 12 and the destination device 14 may be equipped with components for wireless communication. Accordingly, the source device 12 and the destination device 14 may be wireless communication devices.

[0079] The source device 12 and the destination device 14 may install virtual scene application programs (applications, APPs) such as virtual reality (VR) applications, augmented reality (AR) applications, or mixed reality (MR) applications, and may run VR applications, AR applications, or MR applications based on user operations (such as clicks, touches, swipes, shakes, voice controls, etc.). The source device 12 and the destination device 14 may collect images / videos of any object in the environment through a camera and / or sensors, and then display virtual objects on the display device according to the collected images / videos. The virtual objects may be virtual objects in a VR scene, an AR scene, or an MR scene (i.e., objects in a virtual environment).

[0080] It should be noted that in the embodiments of the present application, the virtual scene application programs in the source device 12 and the destination device 14 may be application programs built into the source device 12 and the destination device 14 themselves, or may be application programs provided by third-party service providers installed by the user, and no specific limitation is made thereto.

[0081] In addition, the source device 12 and the destination device 14 may install real-time video transmission applications, such as live broadcast applications. The source device 12 and the destination device 14 may collect images / videos through a camera and then display the collected images / videos on the display device.

[0082] In some cases, Figure 1aThe illustrated video decoding system 10 is merely exemplary, and the techniques provided by embodiments of the present application are applicable to video coding settings (e.g., video encoding or video decoding), which may not necessarily include any data communication between an encoding device and a decoding device. In other examples, data is retrieved from a local memory, sent over a network, etc. A video encoding device may encode data and store the data in a memory, and / or a video decoding device may retrieve data from the memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other but only encode data into the memory and / or retrieve and decode data from the memory.

[0083] Please refer to Figure 1b , Figure 1b which is an exemplary block diagram of a video decoding system 40 provided by an embodiment of the present application. As Figure 1b shown, the video decoding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video codec implemented by processing circuitry 46), an antenna 42, one or more processors 43, one or more memory memories 44, and / or a display device 45.

[0084] As Figure 1b shown, the imaging device 41, the antenna 42, the processing circuitry 46, the video encoder 20, the video decoder 30, the processor 43, the memory memory 44, and / or the display device 45 are capable of communicating with each other. In different instances, the video decoding system 40 may include only the video encoder 20 or only the video decoder 30.

[0085] In some instances, antenna 42 can be used to transmit or receive an encoded bitstream of video data. Additionally, in some instances, display device 45 can be used to present video data. Processing circuitry 46 can include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. Video decoding system 40 can also include optional processor 43, which can similarly include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. Additionally, memory 44 can be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.). In a non-limiting example, memory 44 can be implemented by cache memory. In other instances, processing circuitry 46 can include memory (e.g., cache, etc.) for implementing an image buffer and the like.

[0086] In some instances, video encoder 20 implemented by logic circuitry can include an image buffer (implemented, for example, by processing circuitry 46 or memory 44) and a graphics processing unit (implemented, for example, by processing circuitry 46). The graphics processing unit can be communicatively coupled to the image buffer. The graphics processing unit can include video encoder 20 implemented by processing circuitry 46 to implement various modules discussed with reference to Figure 2 and / or any other encoder system or subsystem described herein. The logic circuitry can be used to perform the various operations discussed herein.

[0087] In some instances, video decoder 30 can be implemented in a similar manner by processing circuitry 46 to implement video decoder 30 discussed with reference to Figure 3 and / or any other decoder system or subsystem described herein. In some instances, video decoder 30 implemented by logic circuitry can include an image buffer (implemented by processing circuitry 46 or memory 44) and a graphics processing unit (implemented, for example, by processing circuitry 46). The graphics processing unit can be communicatively coupled to the image buffer. The graphics processing unit can include video decoder 30 implemented by processing circuitry 46 to implement various modules discussed with reference to Figure 3 and / or any other decoder system or subsystem described herein.

[0088] In some examples, antenna 42 may be used to receive an encoded bitstream of video data. As discussed, the encoded bitstream may include data, indicators, index values, mode selection data, etc. related to the encoded video frames as discussed herein, such as data related to encoded partitions (e.g., transform coefficients or quantized transform coefficients, optional indicators as discussed, and / or data defining the encoded partitions). The video decoding system 40 may further include a video decoder 30 coupled to the antenna 42 and configured to decode the encoded bitstream. A display device 45 is configured to present video frames.

[0089] It should be understood that for the examples described with reference to the video encoder 20 in the embodiments of the present application, the video decoder 30 may be used to perform the reverse process. Regarding signaling syntax elements, the video decoder 30 may be used to receive and parse such syntax elements and accordingly decode the relevant video data. In some examples, the video encoder 20 may entropy encode the syntax elements into an encoded video bitstream. In such examples, the video decoder 30 may parse such syntax elements and accordingly decode the relevant video data.

[0090] For ease of description, the embodiments of the present application are described with reference to the versatile video coding (VVC) reference software or the high-efficiency video coding (HEVC) developed by the video coding experts group (VCEG) of ITU-T and the joint collaboration team on video coding (JCT-VC) of the ISO / IEC motion picture experts group (MPEG). Those of ordinary skill in the art understand that the embodiments of the present application are not limited to HEVC or VVC.

[0091] Encoder and Encoding Method

[0092] As Figure 2 shown, the video encoder 20 includes an input terminal (or input interface) 201, a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy encoding unit 270, and an output terminal (or output interface) 272. The mode selection unit 260 may include an inter prediction unit 244, an intra prediction unit 254, and a partitioning unit 262. The inter prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown).Figure 2 The illustrated video encoder 20 may also be referred to as a hybrid video encoder or a video encoder based on a hybrid video codec.

[0093] See Figure 2 , the inter-frame prediction unit is a trained target model (also referred to as a neural network), and the neural network is used to process an input image or an image region or an image block to generate a predicted value of the input image block. For example, the neural network for inter-frame prediction is used to receive an input image or an image region or an image block and generate a predicted value of the input image or an image region or an image block.

[0094] The residual calculation unit 204, the transform processing unit 206, the quantization unit 208, and the mode selection unit 260 form the forward signal path of the encoder 20, while the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the buffer 216, the loop filter 220, the decoded picture buffer (DPB) 230, the inter-frame prediction unit 244, and the intra-frame prediction unit 254 form the backward signal path of the encoder, where the backward signal path of the encoder 20 corresponds to the signal path of the decoder (see Figure 3 the decoder 30 in). The inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded picture buffer 230, the inter-frame prediction unit 244, and the intra-frame prediction unit 254 also form the "built-in decoder" of the video encoder 20.

[0095] Images and image segmentation (images and blocks)

[0096] The encoder 20 can be used to receive an image (or image data) 17 through the input end 201, etc. For example, an image in an image sequence forming a video or a video sequence. The received image or image data may also be a preprocessed image (or preprocessed image data) 19. For simplicity, the following description uses the image 17. The image 17 may also be referred to as the current image or the image to be encoded (especially when distinguishing the current image from other images in video coding, and other images such as previously encoded and / or decoded images in the same video sequence, i.e., the video sequence including the current image).

[0097] (Digital) images are or can be regarded as two-dimensional arrays or matrices composed of pixel points with intensity values. The pixel points in the array can also be referred to as pixels (pixel or pel, short for picture element). The number of pixel points in the horizontal and vertical directions (or axes) of the array or image determines the size and / or resolution of the image. To represent color, usually three color components are adopted, that is, the image can be represented as or include three pixel point arrays. In the RGB format or color space, the image includes corresponding red, green, and blue pixel point arrays. However, in video coding, each pixel is usually represented in a luminance / chrominance format or color space, such as YCbCr, including a luminance component indicated by Y (sometimes also denoted by L) and two chrominance components denoted by Cb and Cr. The luminance (luma) component Y represents the luminance or gray-level intensity (for example, the two are the same in a grayscale image), while the two chrominance components Cb and Cr represent the chrominance or color information components. Accordingly, an image in the YCbCr format includes a luminance pixel point array of luminance pixel point values (Y) and two chrominance pixel point arrays of chrominance values (Cb and Cr). An image in the RGB format can be converted or transformed into the YCbCr format, and vice versa, and this process is also called color transformation or conversion. If the image is black and white, then the image can only include a luminance pixel point array. Accordingly, the image can be, for example, a luminance pixel point array in a monochrome format or a luminance pixel point array and two corresponding chrominance pixel point arrays in 4:2:0, 4:2:2, and 4:4:4 color formats.

[0098] In one embodiment, an embodiment of the video encoder 20 may include an image segmentation unit ( Figure 2 not shown in the figure) for segmenting the image 17 into a plurality of (usually non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC), or coding tree blocks (CTB), or coding tree units (CTU) in the H.265 / HEVC and VVC standards. The segmentation unit can be used to use the same block size for all images in the video sequence and a corresponding grid defining the block size, or change the block size between images or subsets of images or groups of images, and segment each image into corresponding blocks.

[0099] In other embodiments, the video encoder can be used to directly receive the blocks 203 of the image 17, for example, one, several, or all of the blocks constituting the above-mentioned image 17. The image block 203 can also be referred to as the current image block or the image block to be encoded.

[0100] Similar to Image 17, Image Block 203 is also or can be considered as a two-dimensional array or matrix composed of pixel points with intensity values (pixel values). However, Image Block 203 is smaller than Image 17. In other words, Block 203 can include an array of pixel points (e.g., the luminance array in the case of a monochrome image 17 or the luminance array or chrominance arrays in the case of a color image) or three arrays of pixel points (e.g., one luminance array and two chrominance arrays in the case of a color image 17) or any other number and / or type of arrays according to the color format adopted. The number of pixel points in the horizontal and vertical directions (or axes) of Block 203 defines the size of Block 203. Accordingly, the block can be an array of M×N (M columns × N rows) pixel points, or an array of M×N transform coefficients, etc.

[0101] In one embodiment, Figure 2 The illustrated video encoder 20 is used to encode Image 17 block by block. For example, encoding and prediction are performed on each block 203.

[0102] In one embodiment, Figure 2 The illustrated video encoder 20 can also be used to segment and / or encode an image using slices (also referred to as video slices), where the image can be segmented or encoded using one or more slices (usually non-overlapping). Each slice can include one or more blocks (e.g., Coding Tree Units CTU) or one or more groups of blocks (e.g., coding blocks (tile) in H.265 / HEVC / VVC standards and bricks in the VVC standard).

[0103] In one embodiment, Figure 2 The illustrated video encoder 20 can also be used to segment and / or encode an image using slices / coding block groups (also referred to as video coding block groups) and / or coding blocks (also referred to as video coding blocks), where the image can be segmented or encoded using one or more slices / coding block groups (usually non-overlapping), each slice / coding block group can include one or more blocks (e.g., CTU) or one or more coding blocks, etc., where each coding block can be in a shape such as a rectangle and can include one or more complete or partial blocks (e.g., CTU).

[0104] Residual calculation

[0105] The residual calculation unit 204 is used to calculate the residual block 205 based on the image block (or original block) 203 and the prediction block 265 (the prediction block 265 is introduced in detail later) in the following way: for example, subtract the pixel values of the prediction block 265 from the pixel values of the image block 203 pixel by pixel (pixel by pixel) to obtain the residual block 205 in the pixel domain.

[0106] Transformation

[0107] The transform processing unit 206 is used to perform a discrete cosine transform (DCT) or a discrete sine transform (DST), etc., on the pixel values of the residual block 205 to obtain the transform coefficients 207 in the transform domain. The transform coefficients 207 can also be referred to as transform residual coefficients, representing the residual block 205 in the transform domain.

[0108] The transform processing unit 206 can be used to apply an integer approximation of DCT / DST, such as the transform specified for H.265 / HEVC. Compared with the orthogonal DCT transform, this integer approximation is usually scaled by a certain factor. To maintain the norm of the residual block after the forward and inverse transform processes, other scaling factors are used as part of the transform process. The scaling factors are usually selected according to certain constraints, such as the power of 2 used for shift operations, the bit depth of the transform coefficients, the trade-off between accuracy and implementation cost, etc. For example, at the encoder 20 side, a specific scaling factor is specified for the inverse transform by the inverse transform processing unit 212 (and at the decoder 30 side, a corresponding inverse transform is performed by, for example, the inverse transform processing unit 312), and correspondingly, at the encoder 20 side, a corresponding scaling factor can be specified for the forward transform by the transform processing unit 206.

[0109] In one embodiment, the video encoder 20 (correspondingly, the transform processing unit 206) can be used to output transform parameters such as the type of one or more transforms, for example, directly output or output after being encoded or compressed by the entropy encoding unit 270, such that the video decoder 30 can receive and use the transform parameters for decoding.

[0110] Quantization

[0111] The quantization unit 208 is used to quantize the transform coefficients 207 through, for example, scalar quantization or vector quantization to obtain the quantized transform coefficients 209. The quantized transform coefficients 209 can also be referred to as the quantized residual coefficients 209.

[0112] The quantization process can reduce the bit depth associated with some or all of the transform coefficients 207. For example, during quantization, an n-bit transform coefficient can be rounded down to an m-bit transform coefficient, where n is greater than m. The degree of quantization can be modified by adjusting the quantization parameter (QP). For example, for scalar quantization, different degrees of scaling can be applied to achieve finer or coarser quantization. A smaller quantization step corresponds to finer quantization, while a larger quantization step corresponds to coarser quantization. The appropriate quantization step can be indicated by the quantization parameter (QP). For example, the quantization parameter can be an index of a predefined set of appropriate quantization steps. For example, a smaller quantization parameter can correspond to fine quantization (smaller quantization step), and a larger quantization parameter can correspond to coarse quantization (larger quantization step), and vice versa. Quantization can include dividing by the quantization step, and the corresponding or inverse dequantization performed by the dequantization unit 210 and the like can include multiplying by the quantization step. Embodiments according to some standards such as HEVC can be used to determine the quantization step using the quantization parameter. Generally, the quantization step can be calculated based on the quantization parameter using a fixed-point approximation of an equation involving division. Other scaling factors can be introduced for quantization and dequantization to recover the norm of the residual block that may be modified due to the scaling used in the fixed-point approximation of the equations for the quantization step and the quantization parameter. In one exemplary implementation, the scaling of the inverse transform and dequantization can be combined. Alternatively, a custom quantization table can be used and indicated from the encoder to the decoder in the bitstream. Quantization is a lossy operation, where the larger the quantization step, the greater the loss.

[0113] In one embodiment, the video encoder 20 (correspondingly, the quantization unit 208) can be used to output the quantization parameter (QP), for example, directly output or output after being encoded or compressed by the entropy encoding unit 270, such that the video decoder 30 can receive and use the quantization parameter for decoding.

[0114] Dequantization

[0115] The dequantization unit 210 is used to perform the inverse quantization of the quantization unit 208 on the quantization coefficients to obtain the dequantized coefficients 211. For example, the inverse quantization scheme of the quantization scheme performed by the quantization unit 208 is performed according to or using the same quantization step as the quantization unit 208. The dequantized coefficients 211 can also be referred to as dequantized residual coefficients 211, corresponding to the transform coefficients 207. However, due to the loss caused by quantization, the dequantized coefficients 211 are usually not exactly the same as the transform coefficients.

[0116] Inverse transform

[0117] The inverse transform processing unit 212 is configured to perform an inverse transform of the transform performed by the transform processing unit 206, for example, an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST), to obtain a reconstructed residual block 213 (or corresponding dequantized coefficients 213) in the pixel domain. The reconstructed residual block 213 may also be referred to as the transform block 213.

[0118] Reconstruction

[0119] The reconstruction unit 214 (e.g., an adder 214) is configured to add the transform block 213 (i.e., the reconstructed residual block 213) to the prediction block 265 to obtain a reconstructed block 215 in the pixel domain, for example, adding the pixel values of the reconstructed residual block 213 and the pixel values of the prediction block 265.

[0120] Filtering

[0121] The loop filter unit 220 (or simply referred to as "loop filter" 220) is configured to filter the reconstructed block 215 to obtain a filtered block 221, or generally configured to filter the reconstructed pixels to obtain filtered pixel values. For example, the loop filter unit is used to smoothly perform pixel transitions or improve video quality. The loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination. For example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The order of the filtering process may be a deblocking filter, an SAO filter, and an ALF filter. For another example, a process called luma mapping with chroma scaling (LMCS) (i.e., an adaptive in-loop shaper) is added. This process is performed before deblocking. For another example, the deblocking filtering process may also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Although the loop filter unit 220 is shown as a loop filter in Figure 2 it may be implemented as a post-loop filter in other configurations. The filtered block 221 may also be referred to as the filtered reconstructed block 221.

[0122] In one embodiment, the video encoder 20 (correspondingly, the loop filter unit 220) may be used to output loop filter parameters (such as SAO filter parameters, ALF filter parameters, or LMCS parameters), for example, directly output or output after entropy coding by the entropy coding unit 270, such that the decoder 30 can receive and use the same or different loop filter parameters for decoding.

[0123] Decoded picture buffer

[0124] The decoded picture buffer (DPB) 230 may be a reference image memory that stores reference image data for use by the video encoder 20 when encoding video data. The DPB 230 may be formed of any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of storage devices. The decoded picture buffer 230 may be used to store one or more filtered blocks 221. The decoded picture buffer 230 may also be used to store other previously filtered blocks of the same current image or different images, such as previously reconstructed images, e.g., previously reconstructed and filtered blocks 221, and may provide the complete previously reconstructed i.e., decoded image (and corresponding reference blocks and pixels) and / or partially reconstructed current image (and corresponding reference blocks and pixels), e.g., for inter-frame prediction. The decoded picture buffer 230 may also be used to store one or more unfiltered reconstructed blocks 215, or generally unfiltered reconstructed pixels, e.g., reconstructed blocks 215 that have not been filtered by the loop filter unit 220, or reconstructed blocks or reconstructed pixels that have not undergone any other processing.

[0125] Mode selection (partitioning and prediction)

[0126] The mode selection unit 260 includes a partitioning unit 262, an inter-frame prediction unit 244, and an intra-frame prediction unit 254, and is used to receive or obtain original image data such as the original block 203 (the current block 203 of the current image 17) and reconstructed image data from the decoded picture buffer 230 or other buffers (e.g., a column buffer, Figure 2 not shown in the figure), e.g., filtered and / or unfiltered reconstructed pixels or reconstructed blocks of the same (current) image and / or one or more previously decoded images. The reconstructed image data is used as reference image data required for prediction such as inter-frame prediction or intra-frame prediction to obtain a predicted block 265 or a predicted value 265.

[0127] The mode selection unit 260 can be used to determine or select a segmentation for the current block (including no segmentation) and a prediction mode (e.g., intra-frame or inter-frame prediction mode), generate a corresponding prediction block 265, to calculate the residual block 205 and reconstruct the reconstructed block 215.

[0128] In one embodiment, the mode selection unit 260 can be used to select a segmentation and a prediction mode (e.g., from the prediction modes supported or available to the mode selection unit 260), where the above prediction mode provides the best match or the smallest residual (the smallest residual means better compression in transmission or storage), or provides the smallest signaling overhead (the smallest signaling overhead means better compression in transmission or storage), or considers or balances both of the above at the same time. The mode selection unit 260 can be used to determine the segmentation and the prediction mode according to rate distortion Optimization (RDO), that is, select the prediction mode that provides the smallest rate distortion optimization. The terms "best", "lowest", "optimal", etc. in this article do not necessarily refer to "the best", "the lowest", "the optimal" in general, but can also refer to the situation that meets the termination or selection criteria. For example, values or other limitations that exceed or are lower than the threshold may result in a "sub-optimal selection", but will reduce the complexity and processing time.

[0129] In other words, the segmentation unit 262 can be used to segment the images in the video sequence into a sequence of coding tree units (CTUs), and the CTU 203 can be further segmented into smaller block parts or sub-blocks (forming blocks again). For example, by iteratively using quadtree (QT) partitioning, binary tree (BT) partitioning, triple-tree (TT) partitioning, or any combination thereof, and used to perform predictions on, for example, each of the block parts or sub-blocks, where the mode selection includes selecting the tree structure of the segmented block 203 and selecting the prediction mode applied to each of the block parts or sub-blocks.

[0130] The segmentation (e.g., performed by the segmentation unit 262) and the prediction processing (e.g., performed by the inter-frame prediction unit 244 and the intra-frame prediction unit 254) performed by the video encoder 20 will be described in detail below.

[0131] Segmentation

[0132] The splitting unit 262 can split (or divide) an image block (or CTU) 203 into smaller parts, such as small blocks in the shape of a square or a rectangle. For an image with an array of three pixel points, a CTU consists of an N×N block of luminance pixel points and two corresponding chrominance pixel point blocks. The maximum allowable size of the luminance block in the versatile video coding (VVC) standard under development is specified as 128×128, but it may be specified as a value different from 128×128 in the future, such as 256×256. The CTUs of an image can be grouped / concentrated into slices / coding tree units, coding tree blocks, or tiles. A coding tree block covers a rectangular area of an image, and a coding tree block can be divided into one or more tiles. A tile consists of multiple CTU rows within a coding tree block. A coding tree block that is not divided into multiple tiles can be called a tile. However, a tile is a true subset of a coding tree block and is therefore not called a coding tree block. VVC supports two coding tree unit modes, namely the raster scan slice / coding tree unit mode and the rectangular slice mode. In the raster scan coding tree unit mode, a slice / coding tree unit contains a sequence of coding tree blocks in the raster scan of the coding tree blocks of an image. In the rectangular slice mode, a slice contains multiple tiles of an image, and these tiles together form a rectangular area of the image. The tiles within a rectangular slice are arranged in the tile raster scan order of the slice. These smaller blocks (which can also be called sub-blocks) can be further split into even smaller parts. This is also called tree splitting or hierarchical tree splitting, where a root block at the root tree level 0 (hierarchical level 0, depth 0), etc., can be recursively split into two or more blocks at the next lower tree level, such as nodes at tree level 1 (hierarchical level 1, depth 1). These blocks can in turn be split into two or more blocks at the next lower level, such as tree level 2 (hierarchical level 2, depth 2), etc., until the splitting ends (because an end criterion is met, such as reaching the maximum tree depth or the minimum block size). Blocks that are not further split are also called leaf blocks or leaf nodes of the tree. A tree split into two parts is called a binary-tree (BT), a tree split into three parts is called a ternary-tree (TT), and a tree split into four parts is called a quad-tree (QT).

[0133] For example, a coding tree unit (CTU) may be or include a CTB of luminance pixel points, two corresponding CTBs of chrominance pixel points of an image having a three-pixel-point array, or a CTB of pixel points of a monochrome image or a CTB of pixel points of an image encoded using three independent color planes and a syntax structure (for encoding pixel points). Accordingly, a coding tree block (CTB) may be a block of N×N pixel points, where N may be set to a certain value such that the component is divided into CTBs, and this is segmentation. A coding unit (CU) may be or include a coding block of luminance pixel points, two corresponding coding blocks of chrominance pixel points of an image having a three-pixel-point array, or a coding block of pixel points of a monochrome image or a coding block of pixel points of an image encoded using three independent color planes and a syntax structure (for encoding pixel points). Accordingly, a coding block (CB) may be a block of M×N pixel points, where M and N may be set to a certain value such that the CTB is divided into coding blocks, and this is segmentation.

[0134] For example, in an embodiment, according to HEVC, a coding tree unit (CTU) may be divided into multiple CUs by using a quadtree structure represented as a coding tree. A decision on whether to use inter (temporal) prediction or intra (spatial) prediction to encode an image region is made at the leaf CU level. Each leaf CU may be further divided into one, two, or four PUs according to the PU partition type. The same prediction process is used within one PU, and relevant information is transmitted to the decoder in units of PUs. After obtaining a residual block by applying the prediction process according to the PU partition type, the leaf CU may be segmented into transform units (TUs) according to another quadtree structure similar to the coding tree used for the CU.

[0135] For example, in an embodiment, according to the latest video coding standard currently under development (referred to as Versatile Video Coding (VVC)), a combined quadtree using nested multi-type trees (such as binary trees and ternary trees) is used to divide the segmentation structure for segmenting coding tree units. In the coding tree structure within a coding tree unit, a CU can be square or rectangular. For example, a coding tree unit (CTU) is first segmented by a quadtree structure. A quadtree leaf node is further segmented by a multi-type tree structure. The multi-type tree structure has four partitioning types: vertical binary tree partitioning (SPLIT_BT_VER), horizontal binary tree partitioning (SPLIT_BT_HOR), vertical ternary tree partitioning (SPLIT_TT_VER), and horizontal ternary tree partitioning (SPLIT_TT_HOR). A multi-type tree leaf node is called a coding unit (CU), unless the CU is too large for the maximum transform length, and such a segment is used for prediction and transform processing without any further segmentation. In most cases, this means that the CU, PU, and TU have the same block size in the coding block structure of the quadtree nested multi-type tree. This exception occurs when the maximum supported transform length is less than the width or height of the color component of the CU. VVC has developed a unique signaling mechanism for the segmentation partitioning information in the coding structure with a quadtree nested multi-type tree. In the signaling mechanism, a coding tree unit (CTU), as the root of the quadtree, is first segmented by the quadtree structure. Then each quadtree leaf node (when large enough to be) is further segmented into a multi-type tree structure. In the multi-type tree structure, it is indicated by a first identifier (mtt_split_cu_flag) whether the node is further segmented. When the node is further segmented, a second identifier (mtt_split_cu_vertical_flag) is first used to indicate the partitioning direction, and then a third identifier (mtt_split_cu_binary_flag) is used to indicate whether the partitioning is a binary tree partitioning or a ternary tree partitioning. According to the values of mtt_split_cu_vertical_flag and mtt_split_cu_binary_flag, the decoder can derive the multi-type tree partitioning mode (MttSplitMode) of the CU based on predefined rules or tables. It should be noted that for a certain design, such as the 64×64 luma block and 32×32 chroma pipeline design in a VVC hardware decoder, TT partitioning is not allowed when the width or height of the luma coding block is greater than 64. TT partitioning is also not allowed when the width or height of the chroma coding block is greater than 32. The pipeline design divides the image into multiple virtual pipeline data units (VPDUs), and each VPDU is defined as non-overlapping units in the image. In a hardware decoder, consecutive VPDUs are processed simultaneously in multiple pipeline stages. In most pipeline stages, the VPDU size is roughly proportional to the buffer size, so it is necessary to keep the VPDU small.In most hardware decoders, the VPDU size can be set to the maximum transform block (TB) size. However, in VVC, the splitting of ternary trees (TTs) and binary trees (BTs) may increase the VPDU size.

[0136] In addition, it should be noted that when a part of a tree node block extends beyond the bottom or the right image boundary, the tree node block is forced to be divided until all pixel points of each coded CU are within the image boundary.

[0137] For example, the above intra sub-partitions (ISP) tool can vertically or horizontally divide a luma intra prediction block into two or four sub-parts according to the block size.

[0138] In one example, the mode selection unit 260 of the video encoder 20 can be used to perform any combination of the segmentation techniques described above.

[0139] As described above, the video encoder 20 is used to determine or select the best or optimal prediction mode from a (predetermined) set of prediction modes. The set of prediction modes may include, for example, intra prediction modes and / or inter prediction modes.

[0140] Intra Prediction

[0141] The set of intra prediction modes may include 35 different intra prediction modes. For example, non-directional modes such as DC (or mean) mode and planar mode, or directional modes defined as in HEVC, or may include 67 different intra prediction modes. For example, several traditional angular intra prediction modes are adaptively replaced with wide-angle intra prediction modes for non-square blocks defined in VVC. Also, for example, to avoid the division operation of DC prediction, only the longer side is used to calculate the average value of non-square blocks. And, the intra prediction result of the planar mode can also be modified using the position-dependent intra prediction combination (PDPC) method.

[0142] The intra prediction unit 254 is used to generate an intra prediction block 265 by reconstructing pixel points using adjacent blocks of the same current image according to the intra prediction mode in the set of intra prediction modes.

[0143] The intra-frame prediction unit 254 (or generally the mode selection unit 260) is also used to output intra-frame prediction parameters (or generally information indicating the selected intra-frame prediction mode of the block) in the form of syntax elements 266 to the entropy coding unit 270 for inclusion in the encoded image data 21, so that the video decoder 30 can perform operations, such as receiving and using the prediction parameters for decoding.

[0144] The intra prediction modes in HEVC include DC prediction mode, plane prediction mode and 33 angle prediction modes, with a total of 35 candidate prediction modes. The current block can use the pixels of the reconstructed image blocks on the left and above as references for intra prediction. The image blocks in the surrounding area of the current block used for intra prediction of the current block become reference blocks, and the pixels in the reference blocks are called reference pixels. Among the 35 candidate prediction modes, the DC prediction mode is applicable to the area with flat texture in the current block, and all pixels in this area use the average value of the reference pixels in the reference block as prediction; the plane prediction mode is applicable to image blocks with smoothly changing textures. The current block that meets this condition uses the reference pixels in the reference block for bilinear interpolation as the prediction of all pixels in the current block; the angle prediction mode uses the characteristics that the texture of the current block is highly correlated with the texture of the adjacent reconstructed image blocks, and copies the values of the reference pixels in the corresponding reference block along a certain angle as the prediction of all pixels in the current block.

[0145] The HEVC encoder selects an optimal intra-frame prediction mode for the current block from 35 candidate prediction modes and writes the optimal intra-frame prediction mode into the video bitstream. To improve the coding efficiency of intra-frame prediction, the encoder / decoder derives three most likely modes from the optimal intra-frame prediction modes of the reconstructed image blocks in the surrounding area using intra-frame prediction. If the optimal intra-frame prediction mode selected for the current block is one of the three most likely modes, a first index is encoded to indicate that the selected optimal intra-frame prediction mode is one of the three most likely modes; if the selected optimal intra-frame prediction mode is not one of the three most likely modes, a second index is encoded to indicate that the selected optimal intra-frame prediction mode is one of the other 32 modes (other modes among the 35 candidate prediction modes except the aforementioned three most likely modes). The HEVC standard uses a 5-bit fixed-length code as the aforementioned second index.

[0146] The methods for the HEVC encoder to derive the three most probable modes include: Select the optimal intra prediction modes of the left adjacent picture block and the upper adjacent picture block of the current block and put them into a set. If these two optimal intra prediction modes are the same, only one of them needs to be retained in the set. If these two optimal intra prediction modes are the same and both are angular prediction modes, then select two angular prediction modes adjacent to this angular direction and add them to the set; otherwise, sequentially select the planar prediction mode, the DC mode, and the vertical prediction mode and add them to the set until the number of modes in the set reaches 3.

[0147] After the HEVC decoder performs entropy decoding on the bitstream, it obtains the mode information of the current block. This mode information includes an indication flag indicating whether the optimal intra prediction mode of the current block is among the three most probable modes, and the index of the optimal intra prediction mode of the current block among the three most probable modes or the index of the optimal intra prediction mode of the current block among the other 32 modes.

[0148] Inter-frame prediction

[0149] In a possible implementation, the inter-frame prediction mode set depends on the available reference images (i.e., for example, at least part of the previously decoded images stored in the DBP 230 as described above) and other inter-frame prediction parameters. For example, it depends on whether to use the entire reference image or only a part of the reference image, such as the search window area near the region of the current block, to search for the best matching reference block, and / or for example, it depends on whether to perform pixel interpolation of half pixels, quarter pixels, and / or sixteenth pixels.

[0150] In addition to the above prediction modes, a skip mode and / or a direct mode can also be adopted.

[0151] For example, for extended merge prediction, the merge candidate list for this mode consists of the following five candidate types in order: spatial MVP from spatially adjacent CUs, temporal MVP from collocated CUs, history-based MVP from the FIFO table, pairwise average MVP, and zero MV. A decoder side motion vector refinement (DMVR) based on bipartite matching can be used to increase the accuracy of the MVs of the merge mode. The merge mode with MVD (MMVD) comes from the merge mode with motion vector difference. The MMVD flag is sent immediately after the skip flag and the merge flag to specify whether the CU uses the MMVD mode. A CU-level adaptive motion vector resolution (AMVR) scheme can be used. AMVR supports encoding the MVD of the CU with different precisions. The MVD of the current CU is adaptively selected according to the prediction mode of the current CU. When the CU is encoded in the merge mode, the combined inter / intra prediction (CIIP) mode can be applied to the current CU. The inter and intra prediction signals are weighted and averaged to obtain the CIIP prediction. For affine motion compensation prediction, the affine motion field of the block is described by the motion information of the motion vector with 2 control points (4 parameters) or 3 control points (6 parameters). The subblock-based temporal motion vector prediction (SbTMVP) is similar to the temporal motion vector prediction (TMVP) in HEVC, but predicts the motion vector of the sub-CUs within the current CU. The bi-directional optical flow (BDOF), formerly known as BIO, is a simplified version that reduces computations, especially in terms of the number of multiplications and the size of the multipliers. In the triangular partitioning mode, the CU is evenly partitioned into two triangular parts in two partitioning ways: diagonal partitioning and anti-diagonal partitioning. In addition, the bi-directional prediction mode is extended based on simple averaging to support weighted averaging of two prediction signals.

[0152] The inter prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (both in Figure 2(not shown in the figure). The motion estimation unit can be used to receive or obtain the image block 203 (the current image block 203 of the current image 17) and the decoded image 231, or at least one or more previously reconstructed blocks, for example, the reconstructed blocks of one or more other / different previously decoded images 231, to perform motion estimation. For example, the video sequence may include the current image and the previously decoded image 231, or in other words, the current image and the previously decoded image 231 may be part of the image sequence forming the video sequence or form the image sequence.

[0153] For example, the encoder 20 can be used to select a reference block from multiple reference blocks of the same or different images in multiple other images, and provide the reference image (or reference image index) and / or the offset (spatial offset) between the position (x, y coordinates) of the reference block and the position of the current block as an inter-frame prediction parameter to the motion estimation unit. This offset is also called a motion vector (MV).

[0154] The motion compensation unit is used to obtain, for example, receive, the inter-frame prediction parameter, and perform inter-frame prediction according to or using the inter-frame prediction parameter to obtain the inter-frame prediction block 246. The motion compensation performed by the motion compensation unit may include extracting or generating a prediction block according to the motion / block vector determined by motion estimation, and may also include performing interpolation at sub-pixel accuracy. The interpolation filter can generate pixel points of other pixels from the pixel points of known pixels, thereby potentially increasing the number of candidate prediction blocks available for encoding the image block. Once the motion vector corresponding to the PU of the current image block is received, the motion compensation unit can locate the prediction block pointed to by the motion vector in one of the reference image lists.

[0155] The motion compensation unit can also generate syntax elements related to the block and the video slice for use by the video decoder 30 when decoding the image blocks of the video slice. Additionally, or as an alternative to the slice and the corresponding syntax elements, coded block groups and / or coded blocks and the corresponding syntax elements can be generated or used.

[0156] In the process of obtaining the candidate motion vector list in the advanced motion vector prediction (AMVP) mode, the motion vectors (MVs) that can be added to the candidate motion vector list as an alternative include the MVs of the spatially adjacent and temporally adjacent image blocks of the current block, where the MVs of the spatially adjacent image blocks can include the MVs of the left candidate image block located on the left side of the current block and the upper candidate image block located above the current block. Exemplarily, please refer to Figure 4 , Figure 4 which is an exemplary schematic diagram of the candidate image block provided by the embodiment of the present application, as Figure 4As shown in the figure, the set of left candidate image blocks includes {A0, A1}, the set of upper candidate image blocks includes {B0, B1, B2}, and the set of temporally adjacent candidate image blocks includes {C, T}. All three of these sets can be added as alternatives to the candidate motion vector list. However, according to the existing coding standard, the maximum length of the candidate motion vector list for AMVP is 2. Therefore, it is necessary to determine the MVs of up to two image blocks to be added to the candidate motion vector list from the three sets according to the specified order. This order can be to first consider the set of left candidate image blocks {A0, A1} of the current block (first consider A0, and if A0 is not available, then consider A1), secondly consider the set of upper candidate image blocks {B0, B1, B2} of the current block (first consider B0, if B0 is not available, then consider B1, and if B1 is not available, then consider B2), and finally consider the set of temporally adjacent candidate image blocks {C, T} of the current block (first consider T, and if T is not available, then consider C).

[0157] After obtaining the above candidate motion vector list, the optimal MV is determined from the candidate motion vector list through the rate distortion cost (RDcost), and the candidate motion vector with the minimum RD cost is used as the motion vector predictor (MVP) of the current block. The rate distortion cost is calculated by the following formula:

[0158] J = SAD + λR

[0159] Where J represents the RD cost, SAD is the sum of absolute differences (SAD) between the pixel values of the predicted block obtained by motion estimation using the candidate motion vector and the pixel values of the current block, R represents the bit rate, and λ represents the Lagrange multiplier.

[0160] The encoder transmits the index of the determined MVP in the candidate motion vector list to the decoder. Further, a motion search can be performed within the neighborhood centered on the MVP to obtain the actual motion vector of the current block. The encoder calculates the motion vector difference (MVD) between the MVP and the actual motion vector and also transmits the MVD to the decoder. The decoder parses the index, finds the corresponding MVP in the candidate motion vector list according to the index, parses the MVD, and adds the MVD to the MVP to obtain the actual motion vector of the current block.

[0161] In the process of obtaining the candidate motion information list in the Merge mode, the motion information that can be added to the candidate motion information list as an alternative includes the motion information of the spatially or temporally adjacent image blocks of the current block, where the spatially adjacent image blocks and the temporally adjacent image blocks can be referred to Figure 4 , the candidate motion information corresponding to the spatial domain in the candidate motion information list comes from 5 spatially adjacent blocks (A0, A1, B0, B1, and B2). If the spatially adjacent block is not available or is intra-frame predicted, its motion information is not added to the candidate motion information list. The candidate motion information in the temporal domain of the current block is obtained by scaling the MV of the corresponding position block in the reference frame according to the picture order count (POC) of the reference frame and the current frame. First, it is judged whether the block at position T in the reference frame is available. If it is not available, the block at position C is selected. After obtaining the above candidate motion information list, the optimal motion information is determined from the candidate motion information list through the RD cost as the motion information of the current block. The encoding end transmits the index value of the position of the optimal motion information in the candidate motion information list (denoted as mergeindex) to the decoding end.

[0162] Entropy coding

[0163] The entropy coding unit 270 is used to apply an entropy coding algorithm or scheme (for example, variable length coding (VLC) scheme, context adaptive VLC (CALVC), arithmetic coding scheme, binarization algorithm, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to the quantized residual coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, loop filter parameters, and / or other syntax elements, to obtain encoded image data 21 that can be output in the form of an encoded bitstream 21 through the output end 272, so that a video decoder 30, etc., can receive and use the parameters for decoding. The encoded bitstream 21 can be transmitted to the video decoder 30, or saved in a memory for later transmission or retrieval by the video decoder 30.

[0164] Other structural variants of the video encoder 20 can be used to encode a video stream. For example, a non-transform-based encoder 20 can directly quantize the residual signal in the case where some blocks or frames do not have the transform processing unit 206. In another implementation, the encoder 20 can have a quantization unit 208 and an inverse quantization unit 210 combined into a single unit.

[0165] Decoder and decoding method

[0166] As Figure 3 shown, the video decoder 30 is used to receive encoded image data 21 (e.g., an encoded bitstream 21) encoded by, for example, the encoder 20, to obtain a decoded image 331. The encoded image data or bitstream includes information for decoding the above-mentioned encoded image data, such as data representing image blocks of an encoded video slice (and / or an encoded group of blocks or an encoded block) and related syntax elements.

[0167] In Figure 3 the example of, the decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., an adder 314), a loop filter 320, a decoded image buffer (DBP) 330, a mode application unit 360, an inter prediction unit 344, and an intra prediction unit 354. The inter prediction unit 344 can be or include a motion compensation unit. In some examples, the video decoder 30 can perform a decoding process that is substantially the reverse of the encoding process described with reference to Figure 2 the video encoder 100.

[0168] As in the encoder 20 above, the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded image buffer DPB 230, the inter prediction unit 344, and the intra prediction unit 354 also form the "built-in decoder" of the video encoder 20. Accordingly, the inverse quantization unit 310 can be functionally the same as the inverse quantization unit 110, the inverse transform processing unit 312 can be functionally the same as the inverse transform processing unit 122, the reconstruction unit 314 can be functionally the same as the reconstruction unit 214, the loop filter 320 can be functionally the same as the loop filter 220, and the decoded image buffer 330 can be functionally the same as the decoded image buffer 230. Therefore, the explanations of the corresponding units and functions of the video encoder 20 are correspondingly applicable to the corresponding units and functions of the video decoder 30.

[0169] Entropy decoding

[0170] The entropy decoding unit 304 is used to parse the bitstream 21 (or generally the encoded image data 21) and perform entropy decoding on the encoded image data 21 to obtain quantization coefficients 309 and / or decoded encoded parameters ( Figure 3not shown in [the figure], etc., such as any one or all of inter-frame prediction parameters (e.g., reference image index and motion vector), intra-frame prediction parameters (e.g., intra-frame prediction mode or index), transform parameters, quantization parameters, loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 can be used to apply a decoding algorithm or scheme corresponding to the encoding scheme of the entropy encoding unit 270 of the encoder 20. The entropy decoding unit 304 can also be used to provide inter-frame prediction parameters, intra-frame prediction parameters, and / or other syntax elements to the mode application unit 360, and provide other parameters to other units of the decoder 30. The video decoder 30 can receive video slice and / or video block-level syntax elements. Additionally, or as an alternative to the slice and corresponding syntax elements, coded block groups and / or coded blocks and corresponding syntax elements can be received or used.

[0171] In some implementations, the video decoder 30 can include multiple entropy decoding units 304. For example, in a privacy protection scenario, the bitstream obtained by encoding video images includes a privacy bitstream and a non-privacy bitstream. Therefore, the video decoder 30 can include two entropy decoding units 304. Among them, one entropy decoding unit 304 is used to parse the privacy bitstream in the bitstream, and the other entropy decoder 304 is used to parse the non-privacy bitstream in the bitstream.

[0172] Inverse quantization

[0173] The inverse quantization unit 310 can be used to receive a quantization parameter (QP) (or generally information related to inverse quantization) and quantization coefficients from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304), and perform inverse quantization on the decoded quantization coefficients 309 based on the above quantization parameter to obtain inverse quantization coefficients 311. The inverse quantization coefficients 311 can also be referred to as transform coefficients 311. The inverse quantization process can include using the quantization parameter calculated by the video encoder 20 for each video block in the video slice to determine the quantization degree, and also determine the degree of inverse quantization to be performed.

[0174] Inverse transformation

[0175] The inverse transformation processing unit 312 can be used to receive the dequantized coefficients 311, also referred to as transform coefficients 311, and apply a transformation to the dequantized coefficients 311 to obtain a reconstructed residual block 213 in the pixel domain. The reconstructed residual block 213 can also be referred to as a transform block 313. The transformation can be an inverse transformation, such as inverse DCT, inverse DST, inverse integer transformation, or a conceptually similar inverse transformation process. The inverse transformation processing unit 312 can also be used to receive transform parameters or corresponding information from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304) to determine the transformation applied to the dequantized coefficients 311.

[0176] Reconstruction

[0177] The reconstruction unit 314 (e.g., adder 314) is configured to add the reconstruction residual block 313 to the prediction block 365 to obtain a reconstructed block 315 in the pixel domain, e.g., by adding the pixel values of the reconstruction residual block 313 and the pixel values of the prediction block 365.

[0178] Filtering

[0179] The loop filter unit 320 (in or after the encoding loop) is configured to filter the reconstructed block 315 to obtain a filtered block 321, thereby facilitating pixel conversion or improving video quality, etc. The loop filter unit 320 may include one or more loop filters, such as a deblocking filter, a sample - adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. For example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The order of the filtering process may be a deblocking filter, an SAO filter, and an ALF filter. For another example, a process called luma mapping with chroma scaling (LMCS) (i.e., an adaptive in - loop shaper) is added. This process is performed before deblocking. For another example, the deblocking filtering process may also be applied to internal sub - block edges, such as affine sub - block edges, ATMVP sub - block edges, sub - block transform (SBT) edges, and intra sub - partition (ISP) edges. Although the loop filter unit 320 is shown as a loop filter in Figure 3 it may be implemented as a post - loop filter in other configurations.

[0180] Decoded Picture Buffer

[0181] Subsequently, the decoded video block 321 in an image is stored in the decoded picture buffer 330, and the decoded picture buffer 330 stores a decoded picture 331 as a reference picture, which is used for subsequent motion compensation of other pictures and / or output for display respectively.

[0182] The decoder 30 is configured to output the decoded picture 311 through an output terminal 312, etc., for display to or viewing by a user.

[0183] Prediction

[0184] The inter-frame prediction unit 344 can be functionally the same as the inter-frame prediction unit 244 (especially the motion compensation unit), and the intra-frame prediction unit 354 can be functionally the same as the inter-frame prediction unit 254, and decides on partitioning or segmentation and performs prediction based on the segmentation and / or prediction parameters or corresponding information received from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304). The mode application unit 360 can be used to perform prediction (intra-frame or inter-frame prediction) for each block according to the reconstructed image, block, or corresponding pixel points (filtered or unfiltered), resulting in a predicted block 365.

[0185] When encoding a video slice as an intra-coded (I) slice, the intra-frame prediction unit 354 in the mode application unit 360 is used to generate a predicted block 365 for the image block of the current video slice according to the indicated intra-frame prediction mode and data from the previously decoded blocks of the current image. When the video image is encoded as an inter-coded (i.e., B or P) slice, the inter-frame prediction unit 344 (e.g., the motion compensation unit) in the mode application unit 360 is used to generate a predicted block 365 for the video block of the current video slice according to the motion vector and other syntax elements received from the entropy decoding unit 304. For inter-frame prediction, these predicted blocks can be generated from one of the reference images in one of the reference image lists. The video decoder 30 can use the default construction technique to construct reference frame list 0 and list 1 according to the reference images stored in the DPB 330. In addition to or as an alternative to a slice (e.g., a video slice), the same or similar process can be applied to embodiments of coded block groups (e.g., video coded block groups) and / or coded blocks (e.g., video coded blocks), for example, a video can be encoded using I, P, or B coded block groups and / or coded blocks.

[0186] The mode application unit 360 is used to determine the prediction information for the video block of the current video slice by parsing the motion vector and other syntax elements, and uses the prediction information to generate a predicted block for the current video block being decoded. For example, the mode application unit 360 uses some received syntax elements to determine the prediction mode (e.g., intra-frame prediction or inter-frame prediction) for the video block of the encoded video slice, the inter-frame prediction slice type (e.g., B slice, P slice, or GPB slice), the construction information for one or more reference image lists for the slice, the motion vector for each inter-frame encoded video block of the slice, the inter-frame prediction state for each inter-frame encoded video block of the slice, and other information to decode the video blocks within the current video slice. In addition to or as an alternative to a slice (e.g., a video slice), the same or similar process can be applied to embodiments of coded block groups (e.g., video coded block groups) and / or coded blocks (e.g., video coded blocks), for example, a video can be encoded using I, P, or B coded block groups and / or coded blocks.

[0187] In one embodiment, Figure 3The video encoder 30 can also be used to segment and / or decode an image using slices (also referred to as video slices), where the image can be segmented or decoded using one or more slices (usually non-overlapping). Each slice may include one or more blocks (e.g., CTUs) or one or more groups of blocks (e.g., coding tree units in the H.265 / HEVC / VVC standards and tiles in the VVC standard).

[0188] In one embodiment, Figure 3 The illustrated video decoder 30 can also be used to segment and / or decode an image using slice / coding tree unit groups (also referred to as video coding tree unit groups) and / or coding tree units (also referred to as video coding tree units), where the image can be segmented or decoded using one or more slice / coding tree unit groups (usually non-overlapping), each slice / coding tree unit group may include one or more blocks (e.g., CTUs) or one or more coding tree units, etc., where each coding tree unit can be in a shape such as a rectangle and may include one or more complete or partial blocks (e.g., CTUs).

[0189] Other variants of the video decoder 30 can be used to decode the encoded image data 21. For example, the decoder 30 can produce an output video stream without the loop filter unit 320. For example, a non-transform-based decoder 30 can directly dequantize the residual signal without the inverse transform processing unit 312 for certain blocks or frames. In another implementation, the video decoder 30 can have the dequantization unit 310 and the inverse transform processing unit 312 combined into a single unit.

[0190] It should be understood that in the encoder 20 and the decoder 30, the processing result of the current step can be further processed and then output to the next step. For example, after interpolation filtering, motion vector derivation, or loop filtering, further operations such as clip or shift operations can be performed on the processing result of interpolation filtering, motion vector derivation, or loop filtering.

[0191] It should be noted that further operations can be performed on the derived motion vectors of the current block (including but not limited to the control point motion vectors in the affine mode, the sub-block motion vectors in the affine, planar, ATMVP modes, the temporal motion vectors, etc.). For example, the value of the motion vector can be restricted to a predefined range according to the representation bits of the motion vector. If the representation bits of the motion vector are bitDepth, the range is from -2^(bitDepth - 1) to 2^(bitDepth - 1) - 1, where "^" represents exponentiation. For example, if bitDepth is set to 16, the range is -32768 to 32767; if bitDepth is set to 18, the range is -131072 to 131071. For example, the values of the derived motion vectors (such as the MVs of 4 4×4 sub-blocks in an 8×8 block) are restricted such that the maximum difference between the integer parts of the above 4 4×4 sub-block MVs does not exceed N pixels, for example, does not exceed 1 pixel. Two methods for restricting the motion vector according to bitDepth are provided here.

[0192] Although the above embodiments mainly describe video coding and decoding, it should be noted that the embodiments of the decoding system 10, the encoder 20, and the decoder 30, as well as other embodiments described herein, can also be used for still image processing or coding and decoding, that is, the processing or coding and decoding of a single image independent of any previous or consecutive images in video coding and decoding. Generally, if the image processing is limited to a single image 17, the inter-frame prediction unit 244 (encoder) and the inter-frame prediction unit 344 (decoder) may not be available. All other functions (also referred to as tools or techniques) of the video encoder 20 and the video decoder 30 can equally be used for static image processing, such as residual calculation 204 / 304, transformation 206, quantization 208, dequantization 210 / 310, (inverse) transformation 212 / 312, segmentation 262 / 362, intra-frame prediction 254 / 354, and / or loop filtering 220 / 320, entropy coding 270, and entropy decoding 304.

[0193] Please refer to Figure 5 , Figure 5 which is an exemplary block diagram of the video decoding device 500 provided by the embodiments of the present application. The video decoding device 500 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 500 can be a decoder, such as Figure 1a the video decoder 30 in Figure 1a or an encoder, such as

[0194] Video decoding device 500 includes: an input port 510 (or input port 510) for receiving data and a receiver unit (Rx) 520; a processor, logic unit, or central processing unit (CPU) 530 for processing data; for example, the processor 530 here can be a neural network processor 530; a transmitter unit (Tx) 540 and an output port 550 (or output port 550) for transmitting data; and a memory 560 for storing data. Video decoding device 500 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the input port 510, the receiving unit 520, the transmitting unit 540, and the output port 550 for the exit or entry of optical or electrical signals.

[0195] Processor 530 is implemented by hardware and software. Processor 530 can be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. Processor 530 communicates with the input port 510, the receiving unit 520, the transmitting unit 540, the output port 550, and the memory 560. Processor 530 includes a decoding module 570 (e.g., a neural network-based decoding module 570). Decoding module 570 implements the embodiments disclosed above. For example, decoding module 570 performs, processes, prepares, or provides various encoding operations. Therefore, through decoding module 570, a substantial improvement is provided to the functions of video decoding device 500, and it affects the switching of video decoding device 500 to different states. Alternatively, decoding module 570 is implemented by instructions stored in memory 560 and executed by processor 530.

[0196] Memory 560 includes one or more disks, tape drives, and solid-state drives, and can be used as an overflow data storage device for storing such programs when a selected program is to be executed, and for storing instructions and data read during program execution. Memory 560 can be volatile and / or non-volatile, and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0197] Please refer to Figure 6 , Figure 6An exemplary block diagram of apparatus 600 provided by an embodiment of the present application. Apparatus 600 can be used as Figure 1a either or both of source device 12 and destination device 14 in

[0198] The processor 602 in apparatus 600 can be a central processing unit. Alternatively, processor 602 can be any other type of device or devices, existing or to be developed in the future, capable of manipulating or processing information. Although a single processor such as Figure 6 processor 602 as shown can be used to implement the disclosed implementations, using more than one processor is faster and more efficient.

[0199] In one implementation, the memory 604 in apparatus 600 can be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device can be used as memory 604. Memory 604 can include code and data 606 accessible by processor 602 via bus 612. Memory 604 can also include an operating system 608 and application programs 610, and application programs 610 include at least one program that allows processor 602 to execute the methods described above in this document. For example, application programs 610 can include applications 1 to N, and also include a video decoding application that executes the methods described above in this document.

[0200] Apparatus 600 can also include one or more output devices, such as display 618. In one example, display 618 can be a touch-sensitive display that combines a display with a touch-sensitive element that can be used to sense touch inputs. Display 618 can be coupled to processor 602 via bus 612.

[0201] Although bus 612 in apparatus 600 is described herein as a single bus, bus 612 can include multiple buses. In addition, auxiliary storage can be directly coupled to other components of apparatus 600 or accessed via a network, and can include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Therefore, apparatus 600 can have a variety of configurations.

[0202] As more and more wireless devices are applied to security work, wireless video transmission has increasingly become a topic of our concern. And the real-time transmission of video will inevitably encounter sudden phenomena, which will affect the video picture.

[0203] For example, when the network bandwidth fluctuates, the bit rate of video encoding suddenly decreases, which in turn leads to a decrease in the picture quality and even obvious blocking effects.

[0204] In addition, for example, during the process of collecting video, problems such as camera rotation and sudden changes in picture complexity may also occur, resulting in bitrate fluctuations (i.e., bitrate drops). A typical situation is that suddenly someone passes in front of the camera, which will increase the complexity of the picture. In this case, usually according to the decision of the rate control algorithm, the quantization parameter (QP) of the video is often increased to reduce the bitrate required for the video. Although the bitrate allocated for each frame fluctuates as little as possible under the control of the rate control algorithm, the encoding QP of the video will suddenly increase when the picture complexity increases, which means that the video picture quality starts to decline from this moment, thus leading to a decline in video quality. How to reduce the impact of bitrate fluctuations on video picture quality is an issue that needs to be considered in the encoding and decoding process.

[0205] Currently, in order to reduce the impact of bitrate fluctuations on the encoding quality (picture quality), the video can be encoded with reduced resolution on the basis of hardly changing the encoding QP, and then the resolution can be restored to the original resolution after the network or the picture stabilizes.

[0206] On the one hand, when the resolution is reduced, the number of bits required to encode one frame of the image will decrease; on the other hand, when the complexity of the picture increases and the QP needs to be increased at the same bitrate, the image can be downsampled. For the same number of bits, the quantization step required to encode a large-resolution image is greater than that required to encode a small-resolution image. Therefore, it can be known that after downsampling the image, the resolution of the image can be reduced while the QP is reduced, so as to keep the QP from changing greatly. Compared with the encoding method of increasing the QP, by encoding with reduced resolution, the encoding quality can be improved to a certain extent.

[0207] Some encoders do not support the encoding method where the resolution of the current image to be encoded (the encoded image with reduced resolution) is different from that of the reference image. In this case, if the resolution of the video needs to be switched, an I-frame (IDR frame) needs to be inserted. An I-frame is an image decoded only using intra-frame prediction. The inserted I-frame has the same resolution as the current image to be encoded, so that the I-frame is used as the reference image for the current image to be encoded. However, inserting an I-frame will cause a large fluctuation in the bitrate.

[0208] To address the above problems, in order to reduce the impact of bitrate fluctuations on video quality and avoid the breathing effect caused by inserting I-frames. The embodiments of the present application can modify the motion compensation mode between the reference image and the current image to be encoded, so as to achieve variable-resolution encoding without inserting I-frames.

[0209] Combining the above content, the embodiments of the present application provide an encoding method, which can be applied to the above Figure 1a 、 Figure 1b orFigure 2 the encoder 20 (or video encoder 20) in Figure 7 As shown, the encoding method includes S701 - S702.

[0210] S701. Encode the current image to generate a bitstream, which includes the current encoded image and variable - resolution encoding indication information for indicating the resolution of the current encoded image.

[0211] At the encoding end, if sudden phenomena such as network bandwidth fluctuations, camera movement, or increased picture complexity are detected, the encoding end determines that variable - resolution encoding (i.e., changing the resolution encoding) is required; after the above - mentioned sudden phenomena disappear, the original resolution encoding can be restored. The original resolution is the resolution before the resolution of the image to be encoded changes. Optionally, the original resolution can be the image acquisition resolution.

[0212] Combined with Figure 8 the schematic diagram of the process of the encoding end encoding the image shown, specifically, if the current frame (i.e., the current image, for example, the current image is Figure 8 the input image in

[0213] needs to perform variable - resolution encoding, it is passed through an interpolation filter to resample (such as downsampling) the current image to obtain an image with a changed resolution and sent to the encoder. Figure 8 Continuing to refer to

[0214] During the inter - frame encoding process, for the current encoded block (which can also be called the current block or current encoding unit), during the motion search stage, a reference image with the same resolution as the current frame is used for motion search (i.e., determining the motion vector). During the motion compensation stage, a reference frame with the original resolution is used for inter - frame prediction (i.e., determining the predicted value).

[0215] Specifically, according to the scaling ratio between the current encoded image and the reference image (the ratio of the resolutions of the current encoded image and the reference image, or the ratio of the sizes, or can be called the scaling parameter), the motion vector (MV) of the current block is scaled, and then the predicted value of the current block on the reference frame is obtained through the interpolation filter, and the current interpolation filter supports 2 - fold downsampling in both the horizontal and vertical directions.

[0216] In one implementation, referring toFigure 9 , in the motion compensation part of the current block (current coding unit), (x0, y0) is the starting position of the current block, and xL and yL are the horizontal and vertical step sizes of the pixel to be motion-compensated relative to the starting position, respectively. Then, the position (x ref , y ref ) of the reference pixel of the current pixel on the reference frame can be expressed by the following formula (Formula 1):

[0217]

[0218] wherein, in the formula, scale x and scale y are the ratios of the resolutions of the reference frame and the current frame in the horizontal and vertical directions, respectively (i.e., scaling parameters), mv x and mmv y are the motion vectors MV of the current block in the horizontal and vertical directions, respectively, and the pixel accuracy can be 1 / 16. Finally, the prediction value is calculated using an interpolation filter with 1 / 16 pixel accuracy.

[0219] In the embodiments of the present application, when the value of the variable-resolution coding indication information is the first value, the resolution of the current coded image is equal to 1 / 2 times the original resolution; when the value of the variable-resolution coding indication information is the second value, the resolution of the current coded image is equal to the original resolution.

[0220] In one implementation, when the value of the variable-resolution coding indication information is the first value, the resolution of the current coded image is equal to 1 / 4 times the original resolution; when the value of the variable-resolution coding indication information is the second value, the resolution of the current coded image is equal to the original resolution.

[0221] It can be understood that the resolution of an image is the product of the horizontal size and the vertical size of the image. The horizontal size of the image refers to the horizontal display size of the image, and the vertical size of the image refers to the vertical display size of the image. During the process of encoding an image, the horizontal coding size and the vertical coding size of the image are specified in the encoder. It should be noted that the display size of the image and the coding size of the image are different concepts, that is, the horizontal size and the horizontal coding size are concepts with different meanings, and the vertical size and the vertical coding size are concepts with different meanings.

[0222] In the embodiments of the present application, a flag bit indicating whether variable resolution is enabled for a sequence may be added to a sequence parameter set (SPS). The SPS transmits the horizontal and vertical coding sizes after downsampling, and constraints are imposed on the downsampled coding sizes. A flag bit indicating whether the current coded picture uses 2-fold downsampled coding is added to a picture header (PH). Meanwhile, corresponding constraints are imposed on TMVP (temporal motion vector prediction) during downsampled coding. Table 1 below shows the SPS syntax and semantics.

[0223] Table 1

[0224]

[0225]

[0226] In Table 1, reference_picture_resampling_enable_flag represents the variable resolution coding enable flag, which is a binary variable. A value of '1' indicates that the current sequence allows the use of variable resolution coding; a value of '0' indicates that the current sequence does not allow the use of variable resolution coding. The value of ReferencePictureResamplingEnableFlag is equal to reference_picture_resampling_enable_flag. If reference_picture_resampling_enable_flag does not exist in the bitstream, the value of ReferencePictureResamplingEnableFlag is equal to 0.

[0227] The flag bit reference_picture_resampling_enable_flag is added to the sequence parameter set. If this flag bit is 1, it indicates that the current sequence has enabled variable resolution coding. If this flag bit is 0, it indicates that the current sequence disables variable resolution coding. It should be understood that when reference_picture_resampling_enable_flag is 1, it means that variable resolution coding is allowed. On the premise of allowing variable resolution coding, if variable resolution coding is required for an image, the image is downsampled and coded. If variable resolution coding is not required for the image, the image is directly coded.

[0228] If the flag reference_picture_resampling_enable_flag is 1, the coded width and coded height used for the variable-resolution coding of the downsampled-by-a-factor-of-two image of the current layer are transmitted using sps_rpr_pic_width_in_luma and sps_rpr_pic_height_in_luma respectively.

[0229] In Table 1, sps_rpr_pic_width_in_luma[i] represents the variable-resolution horizontal coding size of the sequence parameter set. sps_rpr_pic_width_in_luma[i], a 14-bit unsigned integer, specifies the coded image width after variable resolution of the image with layer identifier LayerId equal to i (denoted as SpsRprPicWidthInLuma[i]). The value of SpsRprPicWidthInLuma[i] is equal to sps_rpr_pic_width_in_luma[i]. The value of SpsRprPicWidthInLuma[i] cannot be 0 and must be an integer multiple of the minimum coding unit size MinCuSize (e.g., 8×8). It is also specified that the value of SpsRprPicWidthInLuma[i] should be less than or equal to (SpsPicWidthInLuma[i]>>1)+(1<<LcuSizeInBit), where SpsPicWidthInLuma[i] represents the original resolution of the image with LayerId equal to i, and LcuSizeInBit represents the bit width of the largest coding unit. The value of SpsRprPicWidthInLuma[i] should be greater than or equal to (SpsPicWidthInLuma[i]>>1).

[0230] Among them, the symbol ">>" represents a right shift, the symbol "<<" represents a left shift, LcuSizeInBit represents the bit width of the size of the largest coding unit, 1<<LcuSizeInBit (shifting 1 to the left by LcuSizeInBit bits) represents the size of the largest coding unit, SpsPicWidthInLuma[i] represents the sequence parameter set horizontal coding size (the horizontal coding size specified in the SPS, which can be understood as the original horizontal coding size), and SpsPicWidthInLuma[i]>>1 (shifting SpsPicWidthInLuma[i] to the right by 1 bit) represents half of the original horizontal coding size. Thus, it can be known that in the scenario of variable-resolution coding, for the current coded image, the variable-resolution horizontal coding size of the current coded image should be less than or equal to the sum of half of the original horizontal coding size of the current coded image and the horizontal size of the largest coding unit, and greater than or equal to half of the original coding size of the current coded image. It can also be expressed as: (original horizontal coding size of the current coded image / 2) ≤ variable-resolution horizontal coding size of the current coded image ≤ (original horizontal coding size of the current coded image / 2) + horizontal size of the largest coding unit.

[0231] In Table 1, sps_rpr_pic_height_in_luma[i] represents the sequence parameter set variable-resolution vertical coding size, a 14-bit unsigned integer, which specifies the coded image height after variable resolution of the image with LayerId equal to i (denoted as SpsRprPicHeightInLuma[i]). The value of SpsRprPicHeightInLuma[i] is equal to sps_rpr_pic_height_in_luma[i]. The value of SpsRprPicHeightInLuma[i] cannot be 0 and must be an integer multiple of MinCuSize. The value of SpsRprPicHeightInLuma[i] should be less than or equal to (SpsPicHeightInLuma[i]>>1)+(1<<LcuSizeInBit). The value of SpsRprPicHeightInLuma[i] should be greater than or equal to (SpsVerticalSize[i]>>1).

[0232] Among them, SpsPicHeightInLuma[i] represents the vertical coding size of the sequence parameter set (the vertical coding size specified in the SPS, which can be understood as the original vertical coding size), and SpsPicHeightInLuma[i]>>1 (right shift SpsPicHeightInLuma[i] by 1 bit) represents half of the original vertical coding size. Thus, it can be known that in the scenario of variable-resolution coding, for the current coded image, the variable-resolution vertical coding size of the current coded image should be less than or equal to the sum of half of the original vertical coding size of the current coded image and the vertical size of the largest coding unit, and greater than or equal to half of the original vertical coding size of the current coded image. It can also be expressed as: (original vertical coding size of the current coded image / 2) ≤ variable-resolution vertical coding size of the current coded image ≤ (original vertical coding size of the current coded image / 2) + vertical size of the largest coding unit.

[0233] In summary, it can be known that in the embodiments of the present application, the coding size of the current coded image is less than or equal to the sum of half of the original coding size of the current coded image and the size of the largest coding unit, and greater than or equal to half of the original coding size of the current coded image.

[0234] In one implementation, the above variable-resolution coding indication information can be carried in the picture header of the current coded image. Exemplarily, Table 2 shows the syntax and semantics of the picture header.

[0235] Table 2

[0236]

[0237] Among them, picture_rpr_flag represents the above-mentioned variable resolution coding indication information (which can also be called the image variable resolution coding flag), a binary variable. For example, a value of '1' indicates that the horizontal size and vertical size of the current image are equal to 1 / 2 of the sequence parameter set horizontal size SpsHorizontalSize[LayerId] and the sequence parameter set vertical size SpsVerticalSize[LayerId], that is, the horizontal size of the current image is equal to 1 / 2 of the sequence parameter set horizontal size SpsHorizontalSize[LayerId], and the vertical size of the current image is equal to 1 / 2 of the sequence parameter set vertical size SpsVerticalSize[LayerId]. The sequence parameter set horizontal size is the original horizontal size of the image, and the sequence parameter set vertical size is the original vertical size of the image. Further, the horizontal coding size and vertical coding size of the current image are equal to the sequence parameter set variable resolution horizontal coding size SpsRprPicWidthInLuma[LayerId] and the sequence parameter set variable resolution vertical coding size SpsRprPicHeightInLuma[LayerId]. A value of '0' indicates that the horizontal size and vertical size of the current image are equal to the sequence parameter set horizontal size SpsHorizontalSize[LayerId] and the sequence parameter set vertical size SpsVerticalSize[LayerId], and the horizontal coding size and vertical coding size of the current image are equal to the sequence parameter set horizontal coding size SpsPicWidthInLuma[LayerId] and the sequence parameter set vertical coding size SpsPicHeightInLuma[LayerId]. The value of PictureRprFlag is equal to picture_rpr_flag. If picture_rpr_flag does not exist in the bitstream, the value of PictureRprFlag is 0. From the above introduction of picture_rpr_flag, it can be seen that when the value of picture_rpr_flag is 1, it indicates the use of variable resolution coding and the image is downsampled by 1 / 2 in both the horizontal and vertical directions. Since the resolution is the product of the horizontal size and the vertical size, it can be known that picture_rpr_flag indicates the resolution of the image. A value of picture_rpr_flag being 1 also means encoding is performed at 1 / 4 the resolution.

[0238] The horizontal size PicHorizontalSize, vertical size PicVerticalSize, horizontal coding size PicWidthInLuma, and vertical coding size PicHeightInLuma of the current image are obtained as follows:

[0239] PicWidthInLuma = PictureRprFlag? SpsRprPicWidthInLuma[LayerId] : SpsPicWidthInLuma[LayerId];

[0240] PicHeightInLuma = PictureRprFlag? SpsRprPicHeightInLuma[LayerId] : SpsPicHeightInLuma[LayerId];

[0241] PicHorizontalSize = PictureRprFlag? (SpsHorizontalSize[LayerId] >> 1) : SpsHorizontalSize[LayerId];

[0242] PicVerticalSize = PictureRprFlag? (SpsVerticalSize[LayerId] >> 1) : SpsVerticalSize[LayerId].

[0243] In Table 2, picture_tmvp_enable_flag represents the picture temporal motion vector prediction enable flag, a binary variable. A value of '1' indicates that the current picture can use temporal motion vector prediction; a value of '0' indicates that the current picture should not use temporal motion vector prediction. The value of PictureTmvpEnableFlag is equal to the value of picture_tmvp_enable_flag. If picture_tmvp_enable_flag does not exist in the bitstream, the value of PictureTmvpEnableFlag is the value of TmvpEnableFlag.

[0244] Among them, the value of SbTmvpEnableFlag is equal to the value of sbtmvp_enable_flag. If sbtmvp_enable_flag does not exist in the bitstream, the value of SbTmvpEnableFlag is 0. sbtmvp_enable_flag represents the sub-block temporal motion vector prediction enable flag. A value of '1' indicates that sub-block temporal motion vector prediction can be used; a value of '0' indicates that sub-block temporal motion vector prediction should not be used.

[0245] It should be noted that if the current image is a P image (an image that can use the knowledge image and the past images in the display order as reference images for decoding in inter-frame prediction), and the horizontal size or vertical size of the reference image with a reference index value of 0 in the reference image queue 0 is different from the horizontal size or vertical size of the current image, the value of PictureTmvpEnableFlag should be 0. If the current image is a B image (an image that can use the knowledge image, the past and future images in the display order as reference images for decoding in inter-frame prediction), and the horizontal size or vertical size of the reference image with a reference index value of 0 in the reference image queue 1 is different from the horizontal size or vertical size of the current image, the value of PictureTmvpEnableFlag should be 0.

[0246] In the embodiments of the present application, if the variable resolution encoding switch is enabled, it will be transmitted in the image header parameter set whether the current image has the same picture resolution as that in the SPS. If picture_rpr_flag is 1, it means that the current image uses 1 / 2 times resolution encoding (downsampling). If picture_rpr_flag is 0, it means that the current image uses the original resolution encoding.

[0247] In one implementation, according to the description of picture_rpr_flag in the above embodiments, if the variable resolution encoding switch is enabled, it will be transmitted in the image header parameter set whether the current image has the same picture resolution as that in the SPS. If picture_rpr_flag is 1, it means that the current image uses 1 / 4 times resolution encoding (downsampling). If picture_rpr_flag is 0, it means that the current image uses the original resolution encoding.

[0248] In addition, a TMVP switch (i.e., TmvpEnableFlag) is added to the image header, and at the same time, TMVP-related constraints are added, stipulating that TMVP can only be obtained from reference images with the same resolution. Specifically, if the current image is a P image, and the resolution of the reference image with a reference index value of 0 in the reference image queue 0 is different from the current image, picture_tmvp_enable_flag must be 0. If the current image is a B image, and the resolution of the reference image with a reference index value of 0 in the reference image queue 1 is different from the current image, picture_tmvp_enable_flag must be 0.

[0249] It should be noted that when the resolution of the current encoded image is less than the original resolution, the maximum size of the current coding unit is the first size. In the embodiments of the present application, to further reduce the complexity, for the downsampled encoded image, the maximum allowed coding unit is restricted to 64×64, which can enable more simplified and faster encoding in the scenario of resolution reduction encoding. Specifically, when both the width and height of the current coding size are 128, it can only be divided into 4 64×64 by using a quadtree (QT), that is, when both the width and height of the current coding size are 128, it is necessary to use a quadtree (QT) to divide it into 4 64×64. It can be seen that the maximum size of the coding unit is restricted to 64×64.

[0250] In one implementation, the maximum size of the allowed coding unit for the downsampled encoded image can be restricted in the syntax of the coding tree. Table 3 below shows the coding tree syntax and the restrictions.

[0251] Table 3

[0252]

[0253]

[0254]

[0255] In Table 3, qt_split_flag represents the quadtree division flag, which is a binary variable. A value of '1' indicates that the quadtree division should be performed using the division process; a value of '0' indicates that the quadtree division should not be performed. The value of QtSplitFlag is equal to that of qt_split_flag. If qt_split_flag does not exist in the bitstream, the value of QtSplitFlag is equal to the value of allowSplitQt (indicating allowing quadtree division).

[0256] Combined with Table 3, it can be seen that if both the width and height of the maximum coding unit are equal to 128, and the value of PictureRprFlag of this image is 1, then the maximum coding unit should use the quadtree division, that is, the value of qt_split_flag should be '1'.

[0257] S702. Transmit the bitstream.

[0258] After the encoding end completes the encoding, the bitstream (including the encoded image and the above variable resolution encoding indication information) is transmitted to the decoding end, and the decoding end decodes the bitstream and displays the image.

[0259] In summary, in the encoding method provided by the embodiments of the present application, the resolution of an image can be downsampled for encoding, and the resolution of the encoded image is indicated in the bitstream. In this way, in a scenario with bitrate fluctuations, through variable-resolution encoding without increasing the quantization parameter QP, the quality of the video picture can be improved to a certain extent.

[0260] Combined with the description of the encoding process, the embodiments of the present application provide a decoding method, which can be applied to the decoder 30 (or video decoder 30) in the above Figure 1a 、 Figure 1b or Figure 3 . As Figure 10 shown, the decoding method includes S1001-S1002.

[0261] S1001. Parse the bitstream to obtain the variable-resolution encoding indication information (picture_rpr_flag), which is used to indicate the resolution of the current encoded image.

[0262] In the embodiments of the present application, when the value of the variable-resolution encoding indication information is the first value (such as 1), the resolution of the current encoded image is equal to 1 / 2 times the original resolution, indicating that the current encoded image is a downsampled encoded image; when the value of the variable-resolution encoding indication information is the second value (such as 0), the resolution of the current encoded image is equal to the original resolution, indicating that the current encoded image is an encoded image without downsampling, that is, an encoded image with the original resolution.

[0263] It should be understood that the current encoded image (i.e., the residual part) can also be obtained by parsing the bitstream. The current encoded image is the image to be decoded, so it can be called the image to be decoded or the current decoded image.

[0264] S1002. Determine the prediction value of the current coding unit in the current encoded image according to the resolution of the current encoded image and the resolution of the reference image of the current encoded image.

[0265] Wherein, when the resolution of the current encoded image is less than the original resolution, the maximum size of the current coding unit is the first size.

[0266] In the embodiments of the present application, determining the prediction value of the current coding unit in the current encoded image according to the resolution of the current encoded image and the resolution of the reference image of the current encoded image specifically includes: determining the motion vector of the current coding unit, and determining the scaling parameter according to the resolution of the current encoded image and the resolution of the reference image of the current encoded image; and then determining the prediction value of the current coding unit according to the scaling parameter and the motion vector of the current coding unit.

[0267] For the detailed description of S1002, it will be described in the embodiments through different implementation manners (S1102 - S1104 or S1302 - S1304), and will not be elaborated here for the time being.

[0268] In the decoding method provided by the embodiments of the present application, indication information indicating the resolution of the encoded image can be parsed from the bitstream, and prediction of the encoding unit can be performed based on the resolution indicated by the indication information and the resolution of the reference image to complete decoding. This method can, in the scenario of bitrate fluctuation, through variable-resolution encoding, without increasing the quantization parameter QP. Therefore, it can improve the quality of the video picture to a certain extent.

[0269] Combined with the description of the encoding process and S1001 - S102, the embodiments of the present application provide a decoding method, which can be applied to the above Figure 1a 、 Figure 1b or Figure 3 the decoder 30 (or the video decoder 30) in. As Figure 11 shown, this decoding method includes S1101 - S1104.

[0270] S1101. Parse the bitstream to obtain variable-resolution encoding indication information (picture_rpr_flag), which is used to indicate the resolution of the current encoded image.

[0271] In the embodiments of the present application, when the value of the variable-resolution encoding indication information is the first value (such as 1), the resolution of the current encoded image is equal to 1 / 2 times the original resolution, indicating that the current encoded image is a downsampled encoded image; when the value of the variable-resolution encoding indication information is the second value (such as 0), the resolution of the current encoded image is equal to the original resolution, indicating that the current encoded image is an encoded image without downsampling, that is, an encoded image with the original resolution.

[0272] In the embodiments of the present application, when the value of the variable-resolution encoding indication information is the first value (such as 1), the resolution of the current encoded image is equal to 1 / 4 times the original resolution, indicating that the current encoded image is a downsampled encoded image; when the value of the variable-resolution encoding indication information is the second value (such as 0), the resolution of the current encoded image is equal to the original resolution, indicating that the current encoded image is an encoded image without downsampling, that is, an encoded image with the original resolution.

[0273] For more descriptions of the variable-resolution encoding indication information, reference can be made to the relevant descriptions in the embodiments of the above encoding method, and details will not be elaborated here.

[0274] It should be understood that the bitstream can also obtain the current encoded image (i.e., the residual part), and the current encoded image is the image to be decoded. Therefore, it can be called the image to be decoded or the current decoded image. Combined with Figure 12Schematic diagram of the process of decoding an image at the decoding end. Entropy decoding, inverse quantization, and transformation are performed on the residual part to obtain the residual value. After the decoding end obtains the input bitstream, entropy decoding, inverse quantization, and inverse transformation are performed on the bitstream to obtain the residual value of the current decoded image. And when inter-frame prediction is adopted, the decoding end determines the reference image of the current decoded image from the decoding buffer, and determines the current block in the reference image according to the motion vector (MV) (the current block in the reference image refers to the reference block of the current block of the current decoded image in the reference image). Then, it is determined whether the resolutions of the reference image and the current decoded image are the same. If they are the same, sub-pixel interpolation is performed to obtain the predicted value of the current block, thereby obtaining the predicted image of the current decoded image. If they are not the same, motion vector scaling is performed (a detailed description of the motion vector scaling process is given in S1003 below), and then the predicted value of the current block is obtained, and then the predicted image of the current decoded image is obtained.

[0275] S1102. Determine the motion vector of the current coding unit in the current coded image.

[0276] It should be noted that when the resolution of the current coded image is less than the original resolution, the maximum size of the current coding unit is the first size, and the first size can be 64×64.

[0277] In the embodiments of the present application, for the current coding unit, the motion vector of the current coding unit is determined from the motion vector list. It should be noted that the motion vector of the current coding unit determined from the motion vector list is the motion vector of the coding unit at the original resolution. Regarding the determination of the motion vector of the current coding unit (for example, determining the motion vector of the current coding unit from the candidate motion vector list), reference can be made to the prior art materials, and the embodiments of the present application will not elaborate.

[0278] S1103. Scale the motion vector of the current coding unit according to the resolution of the current coded image and the resolution of the reference image of the current coded image.

[0279] Combined with Figure 12 Schematic diagram of the process of decoding an image at the decoding end shown. If the resolution of the reference image is the same as the resolution of the current image, motion compensation is normally performed to obtain the predicted value. If the resolution of the reference image is different from the resolution of the current image, the motion vector MV of the current block (i.e., the current coding unit) is scaled according to the ratio of the resolution of the reference image to the resolution of the current image, and then the current block is projected onto the reference image to obtain the predicted value.

[0280] At the decoding end, for the case where the resolutions of the reference frame and the current frame are different, after completing the decoding of the picture header (obtaining the syntax elements applicable to the picture), it is necessary to update the decoding parameter set according to parameters such as the resolution of the current frame. And MV scaling and interpolation are performed in the inter prediction part.

[0281] In the embodiments of the present application, the above-mentioned scaling of the motion vector MV of the current block (i.e., the current coding unit) according to the ratio of the resolutions of the reference image and the current image specifically includes: determining a scaling parameter according to the ratio of the resolution of the reference image to the resolution of the current image, and then scaling the motion quantity based on the scaling parameter.

[0282] In the embodiments of the present application, the decoding parameter set includes a scaling parameter (the scaling parameter is determined according to the resolution of the reference image and the resolution of the current image), and the scaling parameter includes a scaling flag bit (which can be denoted as scaleFlag) and a scaling offset flag bit (scaleShift). Among them, the scaling flag is used to indicate whether to perform a scaling operation when encoding the current image, and the scaling offset flag bit is used to indicate the number of bits of the scaling offset.

[0283] Exemplarily, when the value of scaleFlag is 0, it means that no scaling is required; when the value of scaleFlag is 1, it means that scaling is required; when the value of scaleShift is 0, it means that the number of bits of the scaling offset is 0; when the value of scaleShift is 1, it means that the number of bits of the scaling offset is 1; when the value of scaleShift is 2, it means that the number of bits of the scaling offset is 2.

[0284] It can be understood that for reference picture list 0 and reference picture list 1, the current picture has corresponding motion vectors. Correspondingly, there are also corresponding scaling flag bits and scaling offset flag bits. For example, for reference picture list 0, the reference frame of the current picture is denoted as refIndexL0, the motion vector can be called the L0 motion vector (denoted as mvE0), the scaling flag bit is denoted as scaleFlagL0, and the scaling offset flag bit is denoted as scaleShiftL0. For reference picture list 1, the reference frame of the current picture is denoted as refIndexL1, the motion vector can be called the L1 motion vector (denoted as mvE1), the scaling flag bit is denoted as scaleFlagL1, and the scaling offset flag bit is denoted as scaleShiftL1. In the embodiments of the present application, the motion information may include the above-mentioned motion vector, scaling flag bit, scaling offset flag bit, reference picture index, and inter prediction mode (or called prediction reference mode).

[0285] Based on the above introduction, since the resolution is the product of the horizontal size and the vertical size, determining the scaling parameter according to the resolution of the reference image and the current image as described above may include: determining the scaling parameter according to the horizontal size of the reference image and the horizontal size of the current image, or determining the scaling parameter according to the vertical size of the reference image and the vertical size of the current image.

[0286] Exemplarily, taking determining the scaling parameter according to the horizontal size of the reference image and the horizontal size of the current image as an example, the scaling parameter is derived in the following manner, that is, scaleFlagL0, scaleFlagL1, scaleShiftL0, and scaleShiftL1 are derived. When the reference image queue of the current image is reference image queue 0, it is denoted as PRED_List0. When the reference image queue of the current image is reference image queue 1, it is denoted as PRED_List1. When the reference image queue of the current image is reference image queue 0 and reference image queue 1, it is denoted as PRED_List01.

[0287] If the prediction reference mode interPredRefMode of the current block is equal to 'PRED_List0' or 'PRED_List01':

[0288] If the horizontal size of the reference image with reference index RefIndexL0 in reference image queue 0 is equal to the horizontal size of the current image, set scaleFlagL0 to 0 and scaleShiftL0 to 1. That is, no scaling operation is performed, and the number of bits of the scaling offset is 1. It can be understood that the horizontal size of the reference image being equal to the horizontal size of the current image includes: the horizontal size of the reference image is equal to the original horizontal size, and the horizontal size of the current image is also equal to the original horizontal size; or, the horizontal size of the reference image is equal to 1 / 2 of the original horizontal size, and the horizontal size of the current image is also equal to 1 / 2 of the original horizontal size.

[0289] Otherwise, if the horizontal size of the reference image with reference index RefIndexL0 in reference image queue 0 is greater than the horizontal size of the current image, set scaleFlagL0 to 1 and scaleShiftL0 to 2. That is, a scaling operation needs to be performed, and the number of bits of the scaling offset is 2. It can be understood that the horizontal size of the reference image being greater than the horizontal size of the current image includes: the horizontal size of the reference image is equal to the original horizontal size, and the horizontal size of the current image is equal to 1 / 2 of the original horizontal size.

[0290] Otherwise, if the horizontal size of the reference image with reference index RefIndexL0 in reference image queue 0 is smaller than the horizontal size of the current image, set scaleFlagL0 to 1 and scaleShiftL0 to 0. That is to say, a scaling operation needs to be performed, and the number of bits of the scaling offset is 0. It can be understood that the horizontal size of the reference image being smaller than the horizontal size of the current image includes: the horizontal size of the reference image being equal to half of the original horizontal size, and the horizontal size of the current image being equal to the original horizontal size.

[0291] If the prediction reference mode interPredRefMode of the current block is equal to 'PRED_List1' or 'PRED_List01':

[0292] If the horizontal size of the reference image with reference index RefIndexL1 in reference image queue 1 is equal to the horizontal size of the current image, set scaleFlagL1 to 0 and scaleShiftL1 to 1;

[0293] Otherwise, if the horizontal size of the reference image with reference index RefIndexL1 in reference image queue 1 is larger than the horizontal size of the current image, set scaleFlagL1 to 1 and scaleShiftL1 to 2;

[0294] Otherwise, if the horizontal size of the reference image with reference index RefIndexL1 in reference image queue 1 is smaller than the horizontal size of the current image, set scaleFlagL1 to 1 and scaleShiftL1 to 0.

[0295] Otherwise (that is, the prediction reference mode interPredRefMode of the current block is not equal to 'PRED_List0' or 'PRED_List01'), set scaleFlagL1 to 0 and scaleShiftL1 to 1.

[0296] Exemplarily, taking the reference image as the image in reference image queue 0 and the pixel precision as 1 / 4, the motion vector is denoted as mvE0, the scaling offset flag scaleFlagL0 = 0, scaleShiftL0 = 1, and the scaled motion vector is mvE0< <scaleshiftl0>>1, where the motion vector in the horizontal direction is denoted as mvE0->x, and the motion vector in the vertical direction is denoted as mvE0->y. After the scaling operation, the motion vector in the horizontal direction is mvE0->x< <scaleshiftl0>>1, the motion vector in the vertical direction becomes mvE0->y after the scaling operation< <scaleshiftl0>> 1。

[0297] For another example, taking the reference image as the image in reference image queue 0 with a pixel precision of 1 / 16, the motion vector is denoted as mvE0, the scaling offset flag scaleFlagL0 = 1, scaleShift L0 = 0, and the scaled motion vector is mvE0 << (scaleShiftL0 + 2) >> 1). Among them, the horizontal motion vector is denoted as mvE0->x, and the vertical motion vector is denoted as mvE0->y. Then, the horizontal motion vector after the scaling operation is mvE0->x << (scaleShiftL0 + 2) >> 1, and the vertical motion vector after the scaling operation is mvE0->y << (scaleShiftL0 + 2) >> 1.

[0298] For another example, taking the reference image as the image in reference image queue 0 with a pixel precision of 1 / 16, the motion vector is denoted as mvE0, the scaling offset flag scaleFlagL0 = 1, scaleShift L0 = 2, and the scaled motion vector is mvE0 << (scaleShiftL0 + 2) >> 1). Among them, the horizontal motion vector is denoted as mvE0->x, and the vertical motion vector is denoted as mvE0->y. Then, the horizontal motion vector after the scaling operation is mvE0->x << (scaleShiftL0 + 2) >> 1, and the vertical motion vector after the scaling operation is mvE0->y << (scaleShiftL0 + 2) >> 1.

[0299] In one implementation, when picture_rpr_flag of the current frame = 1 (resolution reduction is required), first, determine whether the resolution of the reference image (reference frame) is the same as the resolution of the current encoded image (current frame); secondly, when the resolution of the reference image is the same as the resolution of the current encoded image, determine that the scaling factor of the motion vector of the current coding unit is 1 (i.e., no scaling is performed); when the resolution of the reference image is different from the resolution of the current encoded image, for example, when the resolution of the current encoded image is greater than the resolution of the reference frame, the scaling factor of the motion vector of the current coding unit is 2, that is, it is necessary to enlarge the motion vector of the current coding unit; when the resolution of the reference image is different from the resolution of the current encoded image, for example, when the resolution of the current encoded image is less than the resolution of the reference frame, the scaling factor of the motion vector of the current coding unit is 1 / 2, that is, it is necessary to reduce the motion vector of the current coding unit.

[0300] Optionally, since picture_rpr_flag can indicate the resolution of an image, the picture_rpr_flag of the current encoded image and the picture_rpr_flag of the reference image can be compared to determine whether the resolution of the current encoded image is the same as that of the reference image. When the picture_rpr_flag of the current encoded image is the same as the picture_rpr_fla of the reference image, their resolutions are the same; when the picture_rpr_flag of the current encoded image is different from the picture_rpr_flag of the reference image, their resolutions are different.

[0301] Optionally, the resolution of the current encoded image and the resolution of the reference image can also be obtained and compared.

[0302] In one implementation, when the picture_rpr_flag of the current frame = 0 (no resolution downscaling); first, determine the resolution of the reference image (reference frame) and the resolution of the current encoded image (current frame); second, determine whether the resolution of the reference frame is the same as that of the current frame; when the resolution of the reference image is the same as the resolution of the current encoded image, determine that the scaling factor of the motion vector of the current coding unit is 1 (i.e., no scaling); when the resolution of the reference image is different from the resolution of the current encoded image, for example, when the resolution of the current encoded image is less than the resolution of the reference frame, the scaling factor of the motion vector of the current coding unit is 1 / 2, that is, it is necessary to scale down the motion vector of the current coding unit.

[0303] S1104. Determine the predicted value of the current coding unit based on the scaled motion vector.

[0304] According to the scaled motion vector, project the current coding unit onto the reference image to obtain the predicted value of the current coding unit.

[0305] The motion information of the current image includes mvE0 (including mvE0->x and mvE0->y), mvE1 (including mvE1->x and mvE1->y), refIndexL0, refIndexL1, interPredRefMode, the position (xE, yE) of the top-left sample (i.e., pixel) of the current coding unit, x is the horizontal step (offset) of the sample to be predicted relative to the top-left sample, and y is the vertical step (offset) of the sample to be predicted relative to the top-left sample.

[0306] Taking the luminance component as an example, the process of determining the predicted value of a sample to be predicted in the current coding unit is as follows:

[0307] Denote the predicted value of the current coding unit as the prediction sample matrices predMatrixL0 and predMatrixL1.

[0308] If the prediction reference mode interPredRefMode of the current coding unit (i.e., the current block) is equal to 'PRED_List0':

[0309] Then, by way of example, taking reference picture list 0 with a pixel precision of 1 / 4 as an example, if scaleFlagL0 is equal to 0 (no scaling) and scaleShiftL0 = 1, the value of the element (i.e., the sample to be predicted) predMatrixL0[x][y] of the luminance prediction sample matrix predMatrixL0 is the sample value at the position (xPos, yPos) in the 1 / 4 precision luminance sample matrix of the reference picture with the reference index RefIndexL0 in reference picture list 0. Then xPos and yPos are:

[0310] xPos = (((xE + x) << 2) << (scaleShiftL0) >> 1) + (mvE0->x << (scaleShiftL0) >> 1)

[0311] yPos = (((yE + y) << 2) << (scaleShiftL0) >> 1) + (mvE0->y << (scaleShiftL0) >> 1)

[0312] where mvE0->x < <scaleshiftl0>>1 is the scaled motion vector determined in S1103 above.

[0313] When scaleFlagL0 = 0 and scaleShiftL0 = 1, after simplification, it becomes:

[0314] xPos = ((xE + x) << 2) + mvE0->x

[0315] yPos = ((yE + y) << 2) + mvE0->y

[0316] Exemplarily, taking reference picture list 0 with pixel precision of 1 / 16 as an example, when scaleFlagL0 equals 1 and scaleShiftL0 = 0 or 2, the value of the element (i.e., the sample to be predicted) predMatrixL0[x][y] in the luminance prediction sample matrix predMatrixL0 is the sample value at the position (xPos, yPos) in the 1 / 16 precision luminance sample matrix of the reference picture with reference index RefIndexL0 in reference picture list 0.

[0317] Then xPos and yPos are:

[0318] xPos = ((((xE + x) << 2) << (scaleShiftL0 + 2)) >> 1) + (mvE0->x << (scaleShiftL0 + 2) >> 1)

[0319] yPos = ((((yE + y) << 2) << (scaleShiftL0 + 2) >> 1) + (mvE0->y << (scaleShiftL0 + 2) >> 1)

[0320] After simplification, it becomes:

[0321] xPos = ((((xE + x) << 2) + mvE0->x) << (scaleShiftL0 + 2)) >> 1

[0322] yPos = ((((yE + y) << 2) + mvE0->y) << (scaleShiftL0 + 2)) >> 1

[0323] The position (xPos, yPos) of the predicted value of the sample to be predicted in the current coding unit in the reference picture determined above is to determine the sample value at the position (xPos, yPos) in the reference picture as the predicted value of the sample to be predicted.

[0324] Similarly, based on the reference images in the reference image queue 1, the process of determining the prediction value of the current coding unit is similar to the process of determining the prediction value of the current coding unit in the above-mentioned reference image queue 0, and the specific determination process can refer to the above process.

[0325] Based on the residual value of the current coding unit obtained by the above decoding and the prediction value of the current coding unit, the reconstructed value of the current coding unit is obtained, and then the reconstructed image of the current coded image is obtained. After filtering, the decoded image is output.

[0326] Combined with the descriptions of the encoding process, S1001 - S1002, and S1101 - S1104, an embodiment of the present application further provides a decoding method, which can be applied to the above Figure 1a 、 Figure 1b or Figure 3 the decoder 30 (or the video decoder 30) in. As Figure 13 shown, the decoding method includes S1301 - S1304.

[0327] S1301: Parse the bitstream to obtain the variable-resolution coding indication information (picture_rpr_flag), which is used to indicate the resolution of the current coded image.

[0328] For the description of S1301, reference can be made to the descriptions of S1001 and S1101 above, and details are not repeated here.

[0329] S1302: Determine the motion vector of the current coding unit in the current coded image.

[0330] S1303: Determine the scaling parameter according to the resolution of the current coded image and the resolution of the reference image of the current coded image.

[0331] For the description of S1301, reference can be made to the descriptions of S1001 and S1101 above. For the description of S1302, reference can be made to the description of S1102 above. For the description of S1303, reference can be made to the description of the process of determining the scaling parameter in S1103 above, and details are not repeated here.

[0332] S1304: Determine the prediction value of the current coding unit according to the scaling parameter and the motion vector of the current coding unit.

[0333] Based on the scaling parameter scaleFlag and scaleShift, and the motion vector MV in the above embodiments, determine the prediction value of the current coding unit.

[0334] Combined with the description of the above embodiments, the motion information of the current image includes mvE0 (including mvE0->x and mvE0->y), mvE1 (including mvE1->x and mvE1->y), refIndexL0, refIndexL1, interPredRefMode, the position (i.e., pixel) (xE, yE) of the top-left sample of the current coding unit, x is the horizontal step (offset) of the sample to be predicted relative to the top-left sample, and y is the vertical step (offset) of the sample to be predicted relative to the top-left sample.

[0335] Taking the luminance component as an example, the process of determining the predicted value of a sample to be predicted in the current coding unit according to the scaling parameter and the motion vector of the current coding unit is as follows:

[0336] Denote the predicted value of the current coding unit as the prediction sample matrices predMatrixL0 and predMatrixL1.

[0337] If the prediction reference mode interPredRefMode of the current coding unit (i.e., the current block) is equal to 'PRED_List0':

[0338] Then, as an example, taking reference image list 0 with a pixel precision of 1 / 4, if scaleFlagL0 is equal to 0 (no scaling) and scaleShiftL0 = 1, the value of the element (i.e., the sample to be predicted) predMatrixL0[x][y] of the luminance prediction sample matrix predMatrixL0 is the sample value at the position (xPos, yPos) in the 1 / 4 precision luminance sample matrix of the reference image with reference index RefIndexL0 in reference image list 0, where xPos and yPos are:

[0339] xPos = ((xE + x) << 2) + mvE0->x

[0340] yPos = ((yE + y) << 2) + mvE0->y

[0341] As another example, taking reference image list 0 with a pixel precision of 1 / 16, if scaleFlagL0 is equal to 1, the value of the element (i.e., the sample to be predicted) predMatrixL0[x][y] of the luminance prediction sample matrix predMatrixL0 is the sample value at the position (xPos, yPos) in the 1 / 16 precision luminance sample matrix of the reference image with reference index RefIndexL0 in reference image list 0, where xPos and yPos are:

[0342] xPos = ((((xE + x) << 2) + mvE0->x) << (scaleShiftL0 + 2)) >> 1

[0343] yPos = ((((yE + y) << 2) + mvE0->y) << (scaleShiftL0 + 2)) >> 1

[0344] Similarly, based on the reference images in reference image queue 1, the process of determining the prediction value of the current coding unit is similar to the process of determining the prediction value of the current coding unit in the above-mentioned reference image queue 0, and the specific determination process can refer to the above-mentioned process.

[0345] In summary, in the decoding method provided by the embodiments of the present application, the indication information indicating the resolution of the coded image can be parsed from the bitstream, and the coding unit can be predicted based on the resolution indicated by the indication information and the resolution of the reference image to complete decoding. This method can, in a scenario with bitrate fluctuations, through variable-resolution coding, without increasing the quantization parameter QP, and thus can improve the quality of the video picture to a certain extent.

[0346] It can be understood that, in order to implement the above functions, the encoding device and the decoding device include corresponding hardware and / or software modules for executing each function. Combining the algorithm steps of each example described in the embodiments disclosed in this article, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0347] The embodiments of the present application can divide the functional modules of the decoding device according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is illustrative, only a logical function division, and there can be other division methods in actual implementation.

[0348] In the case of dividing each functional module corresponding to each function, Figure 14 shows a possible composition schematic diagram of the encoding device involved in the above embodiment, as Figure 14 shown, the encoding device 1400 may include: an encoding module 1401 and a sending module 1402.

[0349] The encoding module 1401 and the sending module 1402 cooperate to execute the steps of S701 - S702 in the above - mentioned method embodiments.

[0350] In the case of dividing each function into corresponding function modules, Figure 15 The following shows a possible schematic composition diagram of the decoding device involved in the above - mentioned embodiments, as Figure 15 shown, the decoding device 1500 may include: a parsing module 1501 and a prediction module 1502.

[0351] The parsing module 1501 and the prediction module 1502 cooperate to execute the steps of S1001 - S1002, S1101 - S1104, S1301 - S1304 in the above - mentioned method embodiments.

[0352] The embodiments of the present application also provide a chip. Figure 16 The following shows a schematic structural diagram of a chip 1600. The chip 1600 includes one or more processors 1601 and an interface circuit 1602. Optionally, the above - mentioned chip 1600 may further include a bus 1603.

[0353] The processor 1601 may be an integrated circuit chip with signal - processing capabilities. During implementation, each step of the above - mentioned decoding method may be completed by the integrated logic circuit in hardware or instructions in software form in the processor 1601.

[0354] Optionally, the above - mentioned processor 1601 may be a general - purpose processor, a digital signal processing (DSP) processor, an application - specific integrated circuit (ASIC), a field - programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods and steps disclosed in the embodiments of the present application. The general - purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0355] The interface circuit 1602 may be used for sending or receiving data, instructions, or information. The processor 1601 may utilize the data, instructions, or other information received by the interface circuit 1602 for processing, and may send the processed information through the interface circuit 1602.

[0356] Optionally, the chip further includes a memory, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor. A part of the memory may further include a non-volatile random access memory (NVRAM).

[0357] Optionally, the memory stores executable software modules or data structures, and the processor may execute corresponding operations by calling the operation instructions stored in the memory (the operation instructions may be stored in the operating system).

[0358] Optionally, the chip may be used in the encoding device involved in the embodiments of the present application. Optionally, the interface circuit 1602 may be used to output the execution result of the processor 1601. For the decoding method provided by one or more embodiments of the present application, reference may be made to the foregoing embodiments, which will not be elaborated here.

[0359] It should be noted that the functions corresponding to the processor 1601 and the interface circuit 1602 may be implemented by hardware design, may also be implemented by software design, or may be implemented by a combination of software and hardware, which is not limited here.

[0360] Figure 17 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device 1700 may be a processor, a chip, or a functional module in the processor. As Figure 17 shown, the electronic device 1700 includes a processor 1701, a transceiver 1702, and a communication line 1703.

[0361] Among them, the processor 1701 is used to execute any step in the decoding method provided by the embodiments of the present application, and during the execution of any step in the decoding method provided by the embodiments of the present application, the transceiver 1702 and the communication line 1703 may be selectively called to complete the corresponding operations.

[0362] Furthermore, the electronic device 1700 may further include a memory 1704. Among them, the processor 1701, the memory 1704, and the transceiver 1702 may be connected through the communication line 1703.

[0363] Among them, the processor 1701 is a processor, a general-purpose processor, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 1701 may also be other devices with processing functions, such as circuits, devices, or software modules, without limitation.

[0364] The transceiver 1702 is used to communicate with other devices or other communication networks. The other communication networks may be Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc. The transceiver 1702 may be a module, a circuit, a transceiver, or any device capable of implementing communication.

[0365] The transceiver 1702 is mainly used for sending and receiving commands and information, etc. It may include a transmitter and a receiver, which are respectively used for sending and receiving commands and information, etc. Operations other than sending and receiving commands and information, etc. are implemented by the processor.

[0366] The communication line 1703 is used to transmit information between the components included in the electronic device 1700.

[0367] In one design, the processor can be regarded as a logic circuit, and the transceiver can be regarded as an interface circuit.

[0368] The memory 1704 is used to store instructions. Among them, the instructions may be computer programs.

[0369] Among them, the memory 1704 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DRRAM). The memory 1704 can also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, etc. It should be noted that the memory of the systems and methods described herein is intended to include, but not be limited to, these and any other suitable types of memory.

[0370] It should be pointed out that the memory 1704 can exist independently of the processor 1701 or can be integrated with the processor 1701. The memory 1704 can be used to store instructions, program codes, or some data, etc. The memory 1704 can be located inside the electronic device 1700 or outside the electronic device 1700, without limitation. The processor 1701 is used to execute the instructions stored in the memory 1704 to implement the method provided in the above embodiments of the present application.

[0371] In one example, the processor 1701 can include one or more processors, such as Figure 17 processor 0 (CPU0) and processor 1 (CPU1) in

[0372] As an alternative implementation, the electronic device 1700 includes multiple processors. For example, in addition to the processor 1701 in Figure 17 , a processor 1707 may also be included.

[0373] As an alternative implementation, the electronic device 1700 further includes an output device 1705 and an input device 1706. Exemplarily, the input device 1706 is a device such as a keyboard, a mouse, a microphone, or a joystick, and the output device 1705 is a device such as a display screen or a speaker.

[0374] It should be noted that the electronic device 1700 may be a chip system or a device with a similar structure in Figure 17 . The chip system may be composed of chips or may include chips and other discrete devices. Actions, terms, etc. involved among the embodiments of the present application may be referred to each other without limitation. The message names or parameter names in the messages exchanged between the devices in the embodiments of the present application are only examples, and other names may also be adopted in specific implementations without limitation. In addition, Figure 17 the shown composition structure does not constitute a limitation on the electronic device 1700. In addition to the Figure 17 shown components, the electronic device 1700 may include more or fewer components than Figure 17 shown, or combine certain components, or have different component arrangements.

[0375] The processors and transceivers described in the present application may be implemented on an integrated circuit (IC), an analog IC, a radio frequency integrated circuit, a mixed-signal IC, an application specific integrated circuit (ASIC), a printed circuit board (PCB), an electronic device, etc. The processors and transceivers may also be manufactured using various IC process technologies, such as complementary metal oxide semiconductor (CMOS), N-type metal oxide semiconductor (NMOS), P-type metal oxide semiconductor (PMOS), bipolar junction transistor (BJT), BiCMOS, silicon germanium (SiGe), gallium arsenide (GaAs), etc.

[0376] Figure 18 Schematic structural diagram of an image processing apparatus provided by an embodiment of the present application. The image processing apparatus can be applied to the scenarios shown in the above method embodiments. For ease of explanation, Figure 18 only the main components of the image processing apparatus are shown, including a processor 1801, a memory 1802, a control circuit 1803, and an input / output device 1804. The processor 1801 is mainly used for processing communication protocols and communication data, executing software programs, and processing data of software programs. The memory 1802 is mainly used for storing software programs and data. The control circuit 1803 is mainly used for power supply and transmission of various electrical signals. The input / output device 1804 is mainly used for receiving data input by the user and outputting data to the user.

[0377] When the image processing apparatus is the processor 1801, the control circuit 1803 can be a main board, the memory 1802 includes media with storage functions such as a hard disk, RAM, and ROM, the processor 1801 can include a baseband processor 1801 and a central processor. The baseband processor is mainly used for processing communication protocols and communication data, and the central processor is mainly used for controlling the entire image processing apparatus, executing software programs, and processing data of software programs. The input / output device 1804 includes a display screen, a keyboard, a mouse, etc.; the control circuit 1803 can further include or be connected to a transceiver circuit or a transceiver, for example: a network cable interface, etc., for sending or receiving data or signals, such as data transmission and communication with other devices. Further, an antenna can also be included for wireless signal transceiver for data / signal transmission with other devices.

[0378] An embodiment of the present application further provides an encoding apparatus, which includes: at least one processor, when the at least one processor executes program code or instructions, the related method steps are implemented to realize the encoding method in the above embodiment.

[0379] An embodiment of the present application further provides a decoding apparatus, which includes: at least one processor, when the at least one processor executes program code or instructions, the related method steps are implemented to realize the decoding method in the above embodiment.

[0380] Optionally, the apparatus may further include at least one memory for storing the program code or instructions.

[0381] An embodiment of the present application further provides a computer storage medium, in which computer instructions are stored. When the computer instructions run on the decoding apparatus or the encoding apparatus, the decoding apparatus or the encoding apparatus is caused to execute the related method steps to realize the encoding and decoding methods in the above embodiments.

[0382] The embodiments of the present application also provide a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-related steps to implement the encoding and decoding method in the above embodiments.

[0383] The embodiments of the present application also provide a device, which may specifically be a chip, an integrated circuit, a component or a module. Specifically, the device may include a processor connected to a memory for storing instructions, or the device includes at least one processor for obtaining instructions from an external memory. When the device runs, the processor can execute the instructions to enable the chip to execute the encoding and decoding methods in the above method embodiments.

[0384] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the processes or functions in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, a computer, a server or a data center to another website, a computer, a server or a data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, a data center, etc. that includes one or more available media integrated. The available medium may be a magnetic medium (such as a floppy disk, a magnetic disk, a magnetic tape), an optical medium (such as a digital video disc (DVD)), or a semiconductor medium (such as a solid state drive (SSD)), etc.

[0385] Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working processes of the systems, devices and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0386] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections between each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0387] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0388] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0389] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0390] As described above, it is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A decoding method, characterized in that, Comprising: Analyzing the bitstream to obtain variable-resolution coding indication information; The variable-resolution coding indication information is used to indicate the resolution of the current coded image; Determining a prediction value of a current coding unit in the current coded image according to the resolution of the current coded image and the resolution of a reference image of the current coded image; wherein, when the resolution of the current coded image is less than the original resolution, the maximum size of the current coding unit is a first size.

2. The method according to claim 1, characterized in that, The determining the prediction value of the current coding unit according to the resolution of the current coded image and the resolution of the reference image of the current coded image includes: Determining a motion vector of the current coding unit; Scaling the motion vector of the current coding unit according to the resolution of the current coded image and the resolution of the reference image of the current coded image; Determining the prediction value of the current coding unit based on the scaled motion vector.

3. The method according to claim 1, wherein The determining the prediction value of the current coding unit in the current coded image according to the resolution of the current coded image and the resolution of the reference image of the current coded image includes: Determining a motion vector of the current coding unit; Determining a scaling parameter according to the resolution of the current coded image and the resolution of the reference image of the current coded image; Determining the prediction value of the current coding unit according to the scaling parameter and the motion vector of the current coding unit.

4. The method according to any one of claims 1 to 3, wherein The first size is 64×64.

5. The method according to any one of claims 1 to 4, wherein When the value of the variable-resolution coding indication information is a first value, the resolution of the current coded image is equal to 1 / 2 times the original resolution; When the value of the variable-resolution coding indication information is a second value, the resolution of the current coded image is equal to the original resolution.

6. The method according to any one of claims 1 to 4, wherein When the value of the variable-resolution coding indication information is a first value, the resolution of the current coded image is equal to 1 / 4 times the original resolution; When the value of the variable-resolution coding indication information is a second value, the resolution of the current coded image is equal to the original resolution.

7. The method according to any one of claims 1 to 6, wherein The coding size of the current coded image is less than or equal to the sum of half of the original coding size of the current coded image and the size of the maximum coding unit, and greater than or equal to half of the original coding size of the current coded image.

8. A coding method, characterized in that, Comprising: Encoding a current image to generate a bitstream; the bitstream includes a current coded image and variable-resolution coding indication information, and the variable-resolution coding indication information is used to indicate the resolution of the current coded image; wherein, when the resolution of the current coded image is less than the original resolution, the maximum size of the current coding unit is a first size; Transmitting the bitstream.

9. The method according to claim 8, wherein The first size is 64×64.

10. The method according to claim 8 or 9, wherein When the value of the variable-resolution coding indication information is the first value, the resolution of the current coded image is equal to 1 / 2 times the original resolution; When the value of the variable-resolution coding indication information is the second value, the resolution of the current coded image is equal to the original resolution.

11. The method according to claim 8 or 9, wherein When the value of the variable-resolution coding indication information is the first value, the resolution of the current coded image is equal to 1 / 4 times the original resolution; When the value of the variable-resolution coding indication information is the second value, the resolution of the current coded image is equal to the original resolution.

12. The method according to any one of claims 8 to 11, wherein The coding size of the current coded image is less than or equal to the sum of half of the original coding size of the current coded image and the size of the maximum coding unit, and greater than or equal to half of the original coding size of the current coded image.

13. A decoding device, comprising at least one processor and a memory, characterized in that, The at least one processor executes the program or instruction stored in the memory, so that the decoding device implements the method according to any one of claims 1 to 7 above.

14. An encoding device, comprising at least one processor and a memory, characterized in that, The at least one processor executes the program or instruction stored in the memory, so that the encoding implements the method according to any one of claims 8 to 12 above.

15. A computer-readable storage medium for storing a computer program, characterized in that, When the computer program runs on a computer or a processor, the computer or the processor implements the method according to any one of claims 1 to 12 above.

16. A computer program product comprising instructions, characterized in that, When the instruction runs on a computer or a processor, the computer or the processor implements the method according to any one of claims 1 to 12 above.

17. A chip, comprising at least one processor and a memory, characterized in that, The at least one processor executes the program or instruction stored in the memory, so that the chip implements the method according to any one of claims 1 to 12 above.

Images