Encoder, decoder and corresponding inter prediction method

Abstract

An inter prediction method includes obtaining an initial motion vector for a current block, determining a search space position according to the initial motion vector, checking matching costs of the search space position in a checking order to select a target search space position with a minimum matching cost, and determining a corrected motion vector for the current block according to the initial motion vector and the target search space position, wherein a center search space position is checked first in the checking order, and the initial motion vector points to the center search space position.

Description

[0001] This application claims priority to U.S. Provisional Patent Application No. 62 / 791,869, filed January 13, 2019, and U.S. Provisional Patent Application No. 62 / 812,190, filed February 28, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments of this application generally relate to the field of image processing, and more specifically, to inter-frame prediction. Background Technology

[0003] Video decoding (video encoding and decoding) is widely used in digital video applications such as broadcast digital TV, video transmission over the Internet and mobile networks, real-time conversational applications (such as video chat), video conferencing, DVDs and Blu-ray discs, video content capture and editing systems, and cameras for security applications.

[0004] Even with shorter videos, a large amount of video data needs to be described, which can be challenging when the data needs to be transmitted over bandwidth-constrained communication networks or otherwise. Therefore, video data is typically compressed before transmission over modern telecommunications networks. Video size can also be an issue when storing video on storage devices due to potentially limited memory resources. Video compression devices typically use software and / or hardware at the source side to decode the video data before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received at the destination by a video decompression device used to decode the video data. Given limited network resources and the growing demand for higher video quality, there is a need to improve compression and decompression techniques that can increase compression ratios with minimal impact on image quality. Summary of the Invention

[0005] This application provides a scheme for constructing a search space for motion vector correction, and illustrates several checking sequences for checking the matching cost of search space positions in the search space.

[0006] This application provides embodiments of apparatus and methods for encoding and decoding according to the independent claims.

[0007] A first aspect of this application provides an inter-frame prediction method, comprising: obtaining an initial motion vector of a current block; determining a search space position based on the initial motion vector; checking the matching cost of the search space position in an inspection order to select a target search space position with the minimum matching cost; and determining a corrected motion vector of the current block based on the initial motion vector and the target search space position, wherein the center search space position is checked first in the inspection order, and the initial motion vector points to the center search space position.

[0008] In one feasible implementation, the search space location includes the central search space location and adjacent search space locations, wherein determining the search space location based on the initial motion vector includes: determining the central search space location based on the initial motion vector; and determining the adjacent search space locations based on one or more preset offsets and the central search space location.

[0009] In one feasible implementation, the search space is composed of search space locations, and the pattern of the search space is a square of 5×5 search space locations.

[0010] In one feasible implementation, checking the matching cost of the search space locations according to the checking order to select the target search space location with the minimum matching cost includes: sequentially checking the matching cost of each search space location in the search space locations according to the checking order; and selecting the search space location with the minimum matching cost from the search space locations as the target search space location.

[0011] In one feasible implementation, checking the matching cost of each search space location in the search space location in the order of inspection includes: comparing the matching cost of one search space location with the template minimal matching cost.

[0012] When the matching cost of one of the search space locations is less than the minimum matching cost of the template, the matching cost of one of the search space locations is set as the minimum matching cost of the template.

[0013] After the last search space location in the search space is checked, the minimum matching cost of the template is set to the minimum matching cost.

[0014] In one feasible implementation, the central search space position is set to (0, 0) in the coordinate system, with the horizontal direction to the right set as the positive horizontal direction and the vertical direction downward set as the positive vertical direction.

[0015] In one feasible implementation, the inspection sequence is (0,0), (–2,–2), (–1,–2), (0,–2), (1,–2), (2,–2), (–2,–1), (–1,–1), (0,–1), (1,–1), (2,–1), (–2,0), (–1,0), (1,0), (2,0), (–2,1), (–1,1), (0,1), (1,1), (2,1), (–2,2), (–1,2), (0,2), (1,2), (2,2).

[0016] In one feasible implementation, the inspection sequence is (0,0), (–1,0), (0,1), (1,0), (0,–1), (–1,–1), (–1,1), (1,1), (1,–1), (–2,0), (–2,1), (–2,2), (–1,2), (0,2), (1,2), (2,2), (2,1), (2,0), (2,–1), (2,–2), (1,–2), (0,–2), (–1,–2), (–2,–2), (–2,–1).

[0017] In one feasible implementation, the inspection sequence is (0,0), (-1,0), (0,1), (1,0), (0,-1), (-1,-1), (-1,1), (1,1), (1,-1), (-2,0), (0,2), (2,0), (0,-2), (-2,-1), (-2,1), (-2,2), (-1,2), (1,2), (2,2), (2,1), (2,-1), (2,-2), (1,-2), (-1,-2), (-2,-2).

[0018] In one feasible implementation, the inspection sequence is (0,0), (-1,0), (0,1), (1,0), (0,-1), (-1,-1), (-1,1), (1,1), (1,-1), (-2,0), (0,2), (2,0), (0,-2), (-2,-2), (-2,2), (2,2), (2,-2), (-2,-1), (-2,1), (-1,2), (1,2), (2,1), (2,-1), (1,-2), (-1,-2).

[0019] A second aspect of this application provides an inter-frame prediction apparatus, comprising: an acquisition module for acquiring an initial motion vector of a current block; a setting module for determining a search space position based on the initial motion vector; a calculation module for checking the matching cost of the search space position in an inspection order to select a target search space position with the minimum matching cost; and a prediction module for determining a corrected motion vector of the current block based on the initial motion vector and the target search space position, wherein the center search space position is checked first in the inspection order, and the initial motion vector points to the center search space position.

[0020] In one feasible implementation, the search space location includes the central search space location and adjacent search space locations, wherein the setting module is used to: determine the central search space location based on the initial motion vector; and determine the adjacent search space locations based on one or more preset offsets and the central search space location.

[0021] In one feasible implementation, the search space is composed of search space locations, and the pattern of the search space is a square of 5×5 search space locations.

[0022] In one feasible implementation, the calculation module is used to: sequentially check the matching cost of each search space location in the search space according to the checking order; and select the search space location with the minimum matching cost from the search space locations as the target search space location.

[0023] In one feasible implementation, the calculation module is used to: compare the matching cost of one search space location in the search space with the minimum matching cost of the template; when the matching cost of one search space location in the search space is less than the minimum matching cost of the template, set the matching cost of one search space location in the search space as the minimum matching cost of the template.

[0024] After checking the last search space position in the search space, the minimum matching cost of the template is set to the minimum matching cost.

[0025] In one feasible implementation, the central search space position is set to (0, 0) in the coordinate system, with the horizontal direction to the right set as the positive horizontal direction and the vertical direction downward set as the positive vertical direction.

[0026] In one feasible implementation, the inspection sequence is (0,0), (–2,–2), (–1,–2), (0,–2), (1,–2), (2,–2), (–2,–1), (–1,–1), (0,–1), (1,–1), (2,–1), (–2,0), (–1,0), (1,0), (2,0), (–2,1), (–1,1), (0,1), (1,1), (2,1), (–2,2), (–1,2), (0,2), (1,2), (2,2).

[0027] In one feasible implementation, the inspection sequence is (0,0), (–1,0), (0,1), (1,0), (0,–1), (–1,–1), (–1,1), (1,1), (1,–1), (–2,0), (–2,1), (–2,2), (–1,2), (0,2), (1,2), (2,2), (2,1), (2,0), (2,–1), (2,–2), (1,–2), (0,–2), (–1,–2), (–2,–2), (–2,–1).

[0028] In one feasible implementation, the inspection sequence is (0,0), (-1,0), (0,1), (1,0), (0,-1), (-1,-1), (-1,1), (1,1), (1,-1), (-2,0), (0,2), (2,0), (0,-2), (-2,-1), (-2,1), (-2,2), (-1,2), (1,2), (2,2), (2,1), (2,-1), (2,-2), (1,-2), (-1,-2), (-2,-2).

[0029] In one feasible implementation, the inspection sequence is (0,0), (-1,0), (0,1), (1,0), (0,-1), (-1,-1), (-1,1), (1,1), (1,-1), (-2,0), (0,2), (2,0), (0,-2), (-2,-2), (-2,2), (2,2), (2,-2), (-2,-1), (-2,1), (-1,2), (1,2), (2,1), (2,-1), (1,-2), (-1,-2).

[0030] A third aspect of this application provides a method for determining motion vectors for inter-frame prediction of a current block in a video image (or frame), the method comprising: acquiring an initial motion vector; acquiring at least two sets of points based on the initial motion vector (e.g., a set of points may include only one point); and obtaining the motion vector of the current block based on the at least two sets of points and a cost function.

[0031] In one feasible implementation, the at least two sets of points include all points within a square centered on the point corresponding to the initial motion vector.

[0032] In one feasible implementation, the at least two sets of points include all points within a square, the corner points of which are determined by coordinates (–2, –2), (2, 2), (–2, 2), (2, –2).

[0033] In one feasible implementation, the first set of points in the at least two sets of points includes a center point, and the initial motion vector points to the center point.

[0034] In one feasible implementation, the second set of points in the at least two sets of points includes four adjacent points to the left, top, right and bottom of the center point, and the initial motion vector points to the center point.

[0035] In one feasible implementation, the third group of points in the at least two groups of points includes four points, which are one pixel sample away from the center point and are not included in the second group.

[0036] In one feasible implementation, the fourth set of points in the at least two sets of points includes points that are at least 2 pixel samples away from the center point on one coordinate axis.

[0037] In one feasible implementation, when processing the at least two sets of points using a cost function, the at least two sets of points are sorted.

[0038] In one feasible implementation, the fourth set of points in the at least two sets of points includes points that are 2 pixel samples away from the center point on one coordinate axis and 0 pixel samples away from the center point on another coordinate axis.

[0039] In one feasible implementation, the fifth set of points in the at least two sets of points includes points that are at least 2 pixel samples away from the center point on one coordinate axis and are not included in the fourth set of points.

[0040] In one feasible implementation, the fifth set of points in the at least two sets of points includes points that are 2 pixel samples away from the center point on both the –x coordinate axis and the –y coordinate axis.

[0041] In one feasible implementation, the sixth set of points in the at least two sets of points includes points that are at least 2 pixel samples away from the center point on one coordinate axis and are not included in the fourth set of points or the fifth set of points.

[0042] In one feasible implementation, the fourth set of points in the at least two sets of points includes points that are 2 pixel samples away from the center point on one coordinate axis and 0 pixel samples away from the center point on the other coordinate axis.

[0043] In one feasible implementation, when one of the at least two sets of points includes at least two points, the set of points is sorted according to a predefined rule.

[0044] In one feasible implementation, when one of the at least two sets of points includes at least two points, the leftmost point in that set of points is selected as the first point in that set.

[0045] In one feasible implementation, when one of the at least two sets of points includes at least two points, and when there are multiple points that are the leftmost points in the set, the leftmost point in the set that has a coordinate displacement closer to 0 in the vertical direction (–y direction) is selected as the first point in the set.

[0046] In one feasible implementation, when one of the at least two sets of points includes at least two points, and when multiple points are the leftmost points in that set, if the points have the same displacement in the –y direction, then the leftmost point in the top direction is selected as the first point of that set of points.

[0047] In one feasible implementation, when one of the at least two sets of points includes at least two points, after determining the first point in the set, the remaining points in the same set are sorted based on a clockwise or counterclockwise scan of the points surrounding the center point.

[0048] In one feasible implementation, the initial motion vector corresponds to a motion vector derived using an index indicated in the bitstream.

[0049] In one feasible implementation, the initial motion vector and the acquired motion vector are not decoded into the bitstream.

[0050] The fourth aspect of this application provides a computer program product including program code, which, when executed by a computer or processor, is used to perform the method according to the first or third aspect.

[0051] A fifth aspect of this application provides a decoder comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the processors and storing a program executed by the processors, wherein, when the processors execute the program, the decoder performs a method according to a first or third aspect.

[0052] A sixth aspect of this application provides an encoder comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the processors and storing a program executed by the processors, wherein, when the processors execute the program, the encoder performs a method according to a first aspect or a third aspect.

[0053] A seventh aspect of this application provides a non-transitory computer-readable medium carrying program code, which, when executed by a computer device, causes the computer device to perform the method of the first or third aspect.

[0054] The above and other objectives are achieved through the subject matter claimed in the independent claims. Other implementations are apparent from the dependent claims, the specification, and the drawings.

[0055] Specific embodiments are summarized in the appended independent claims, and other embodiments are summarized in the dependent claims.

[0056] One or more embodiments will be described in detail in the accompanying drawings and the following description. Other features, objectives, and advantages will be apparent from the specification, drawings, and claims. Attached Figure Description

[0057] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0058] Figure 1A This is a block diagram illustrating an example of a video decoding system used to implement embodiments of this application.

[0059] Figure 1B This is a block diagram of another example of a video decoding system used to implement embodiments of this application.

[0060] Figure 2 This is a block diagram of an example video encoder used to implement embodiments of this application.

[0061] Figure 3 This is a block diagram of an example structure for implementing the video decoder of the embodiments of this application.

[0062] Figure 4 A block diagram of an example encoding or decoding device.

[0063] Figure 5 A block diagram for another example of an encoding or decoding device.

[0064] Figure 6 A block diagram for another example of a prediction method.

[0065] Figure 7 A diagram illustrating the inspection sequence.

[0066] Figure 8Another illustration of the inspection sequence.

[0067] Figure 9 Another illustration of the inspection sequence.

[0068] Figure 10 Another illustration of the inspection sequence.

[0069] Figure 11 Another illustration of the inspection sequence.

[0070] Figure 12 Another illustration of the inspection sequence.

[0071] Figure 13 Another illustration of the inspection sequence.

[0072] Figure 14 Another illustration of the inspection sequence.

[0073] Figure 15 This is a block diagram of an example of a prediction apparatus for implementing embodiments of this application.

[0074] Figure 16 A block diagram of an example encoding or decoding device.

[0075] Figure 17 A block diagram of an exemplary structure for a content delivery system 3100 that implements a content distribution service.

[0076] Figure 18 A block diagram of an example structure for a terminal device.

[0077] In the following text, unless otherwise expressly specified, the same reference numerals refer to the same or at least functionally equivalent features. Detailed Implementation

[0078] In the following description, reference is made to the accompanying drawings, which form part of this application, illustrating by way of description specific aspects of embodiments of the present application or specific aspects in which embodiments of the invention may be used. It should be understood that embodiments of the present application can be used in other aspects and include structural or logical variations not depicted in the drawings. Therefore, the following detailed description is not intended to be limiting, and the scope of this application is defined by the appended claims.

[0079] For example, it should be understood that the disclosure of the described method also applies to the corresponding device or system used to perform the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units, such as functional units for performing the described one or more method steps (e.g., a unit performing one or more steps; or multiple units, each performing one or more of the multiple steps), even if the one or more units are not explicitly described or shown in the drawings. Furthermore, for example, if a particular apparatus is described according to one or more units (e.g., functional units), the corresponding method may include a step for performing the function of one or more units (e.g., a step performing the function of one or more units, or multiple steps, each performing the function of one or more of the multiple units), even if the one or more steps are not explicitly described or shown in the drawings. Furthermore, it should be noted that, unless specifically indicated, features of the various exemplary embodiments and / or aspects described herein can be combined with each other.

[0080] Video decoding generally refers to the processing of a sequence of images that constitute a video or video sequence. In the field of video decoding, the term "frame" can be used instead of the term "picture" or "image." Video decoding (or decoding as commonly referred to) consists of two parts: video encoding and video decoding. Video encoding is performed at the source end and typically involves processing (e.g., by compression) the raw video image to reduce the amount of data required to represent the video image (for more efficient storage and / or transmission). Video decoding is performed at the destination end and typically involves inverse processing relative to the encoder to reconstruct the video image. The "decoding" of the video image (or generally referred to as an image) involved in the embodiments should be understood as the "encoding" or "decoding" of the video image or the respective video sequence. The combination of the encoding and decoding parts is also called a codec (encoding and decoding, CODEC).

[0081] In lossless video decoding, the original video image can be reconstructed, meaning the reconstructed video image has the same quality as the original (assuming no transmission loss or other data loss during storage or transmission). In lossy video decoding, further compression is performed through methods such as quantization to reduce the amount of data required to represent the video image. As a result, the video image cannot be fully reconstructed at the decoder side; that is, the quality of the reconstructed video image is lower or worse than the quality of the original video image.

[0082] Several video decoding standards belong to the "lossy hybrid video coding and decoding" group (i.e., combining spatial and temporal predictions in the sample domain with 2D transform decoding for quantization in the transform domain). Each image in a video sequence is typically divided into a set of non-overlapping blocks, and decoding is usually performed at the block level. In other words, on the encoder side, the video is typically processed (i.e., encoded) at the block (video block) level, for example, by generating prediction blocks using spatial (intra-frame image) prediction and / or temporal (inter-frame image) prediction, subtracting the prediction block from the current block (currently processed / to-be-processed block) to obtain a residual block, transforming the residual block, and quantizing the residual block in the transform domain to reduce the amount of data to be sent (compressed). On the decoder side, the inverse processing relative to the encoder is used to encode or compress blocks to reconstruct the current block for representation. Furthermore, the encoder processes according to the decoder processing loop, such that both generate the same predictions (e.g., intra-frame and inter-frame predictions) and / or reconstructions for processing, i.e., decoding subsequent blocks.

[0083] In the following embodiment of the video decoding system 10, the video encoder 20 and the video decoder 30 are based on Figures 1A to 3 Describe it.

[0084] Figure 1A This is a schematic block diagram of an exemplary decoding system 10, such as a video decoding system 10 (or simply decoding system 10) that can use the techniques of this application. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) of the video decoding system 10 represent examples of devices that can be used to perform the techniques according to the various examples described in this application.

[0085] like Figure 1A As shown, the decoding system 10 includes a source device 12, which provides encoded image data 21 to a destination device 14 or the like for decoding the encoded data 13.

[0086] The source device 12 includes an encoder 20 and may additionally (optionally) include an image source 16, a preprocessor (preprocessing unit) 18 (e.g., an image preprocessor 18), and a communication interface or communication unit 22.

[0087] Image source 16 may include or may be: any type of image capture device, such as a camera for capturing real-world images; and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images; or any type of device for acquiring and / or providing real-world images, computer-animated images (e.g., screen content, virtual reality (VR) images), and / or any combination thereof (e.g., augmented reality (AR) images). Image source may be any type of memory / storage for storing any of the images described above.

[0088] In order to distinguish it from the preprocessor 18 and the processing performed by the preprocessing unit 18, the image or image data 17 may also be referred to as the raw image or raw image data 17.

[0089] The preprocessor 18 receives (raw) image data 17 and preprocesses the image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. The preprocessing performed by the preprocessor 18 may include trimming, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It is understood that the preprocessing unit 18 may be an optional component.

[0090] Video encoder 20 is used to receive preprocessed image data 19 and provide encoded image data 21 (e.g., according to the following). Figure 2 (Further detailed description).

[0091] The communication interface 22 in the source device 12 can be used to: receive encoded image data 21 and send the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device via the communication channel 13 for storage or direct reconstruction.

[0092] Destination device 14 includes decoder 30 (e.g., video decoder 30) and may additionally (optionally) include communication interface or communication unit 28, post-processor 32 (or post-processing unit 32) and display device 34.

[0093] The communication interface 28 of the destination device 14 is used to receive encoded image data 21 (or any other processed version) directly from the source device 12, or from any other source (e.g., a storage device, such as an encoded image data storage device), and to provide the encoded image data 21 to the decoder 30.

[0094] Communication interfaces 22 and 28 can be used to send or receive encoded image data 21 or encoded data 13 via a direct communication link (e.g., a direct wired or wireless connection) between source device 12 and destination device 14, or via any type of network (e.g., wired network, wireless network, any combination of wired and wireless networks, any type of private and public network, any type of private and public network).

[0095] For example, the communication interface 22 can be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission encoding or processing, so that it can be transmitted on a communication link or communication network.

[0096] Communication interface 28 corresponds to communication interface 22. For example, it can be used to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or decapsulation process to obtain encoded image data 21.

[0097] Both communication interface 22 and communication interface 28 can be configured as follows: Figure 1A The communication channel 13, from source device 12 to destination device 14, is a one-way communication interface indicated by the arrow, or configured as a two-way communication interface, and can be used to send and receive messages, establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission (e.g., encoded image data transmission).

[0098] Decoder 30 is used to receive encoded image data 21 and provide decoded image data 31 or decoded image 31 (e.g., according to the following). Figure 3 or Figure 5 (Further detailed description).

[0099] The post-processor 32 of the destination device 14 is used to post-process the decoded image data 31 (also known as reconstructed image data), such as the decoded image 31, to obtain post-processed image data 33, such as the post-processed image 33. The post-processing performed by the post-processing unit 32 may include color format conversion (e.g., from YCbCr to RGB), color correction, trimming or resampling, or any other processing, to prepare the decoded image data 31 for display by a display device 34, etc.

[0100] The display device 34 of the destination device 14 is used to receive post-processed image data 33 for displaying the image, for example, to a user or viewer. The display device 34 can be or includes any type of display for presenting the reconstructed image, such as an integrated or external display or monitor. For example, the display can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, liquid crystal on silicon (LCOS), a digital light processor (DLP), or any other type of display.

[0101] although Figure 1A Source device 12 and destination device 14 are described as devices, but embodiments of the devices may also include both devices or the functions of both devices, i.e., including the functions of source device 12 or its corresponding functions and the functions of destination device 14 or its corresponding functions. In such embodiments, the functions of source device 12 or its corresponding functions and the functions of destination device 14 or its corresponding functions may be implemented by the same hardware and / or software, by separate hardware and / or software, or by any combination thereof.

[0102] Based on the above description, it will be obvious to those skilled in the art that, Figure 1A The presence and (precise) functional division of the different units or functions of the source device 12 and / or destination device 14 shown may vary depending on the actual device and application.

[0103] Encoder 20 (e.g., video encoder 20) or decoder 30 (e.g., video decoder 30) or both can be transmitted via, for example, Figure 1B The processing circuitry shown may be implemented using one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video decoding processors, or any combination thereof. Encoder 20 may be implemented using processing circuitry 46 to achieve a combination of... Figure 2 The encoder 20 and / or any other encoder system or subsystem described herein, including the various modules discussed. The decoder 30 may be implemented via processing circuitry 46 to achieve integration. Figure 3The decoder 30 is one of the various modules discussed herein and / or any other decoder system or subsystem described herein. Processing circuitry can be used to perform the various operations described herein. For example... Figure 5 As shown, if these techniques are partially implemented in software, the device can store the software instructions in a suitable non-transitory computer-readable storage medium, and these instructions can be executed in hardware by one or more processors to perform the techniques of the present invention. Either the video encoder 20 and the video decoder 30 can be integrated into a single device as part of a combined codec (encoder / decoder, CODEC), such as... Figure 1B As shown.

[0104] Source device 12 and destination device 14 can include any of a variety of devices, including any type of handheld or fixed device, such as laptops or tablets, mobile phones, smartphones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, video game consoles, video streaming devices (such as content service servers or content distribution servers), broadcast receiver devices, broadcast transmitter devices, etc., and may or may not use any type of operating system. In some cases, source device 12 and destination device 14 can be used for wireless communication. Therefore, source device 12 and destination device 14 can be wireless communication devices.

[0105] In some cases, Figure 1A The video decoding system 10 shown is merely an example. The technology of this application can be applied to video decoding setups (e.g., video encoding or video decoding) and does not necessarily include any data communication between the encoding and decoding devices. In other examples, data may be retrieved from local memory, streamed over a network, etc. The video encoding device may encode the data and store it in memory, and / or the video decoding device may retrieve and decode the data from memory. In some examples, encoding and decoding are performed by devices that do not communicate with each other but simply encode data into memory and / or retrieve and decode data from memory.

[0106] For ease of description, embodiments of this application are described, for example, with reference to the high-efficiency video coding (HEVC), versatile video coding (VVC) reference software, and next-generation video coding standards developed by the joint collaboration team on video coding (JCT-VC) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG). Those skilled in the art will understand that embodiments of this application are not limited to HEVC or VVC.

[0107] Encoders and Encoding Methods

[0108] Figure 2 This is a schematic block diagram of an exemplary video encoder 20 used to implement the technology of this application. Figure 2 In the example, the video encoder 20 includes an input terminal 201 (or input interface 201), a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output terminal 272 (or output interface 272). The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254, and a partitioning unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown in the figure). According to the hybrid video codec, Figure 2 The video encoder 20 shown can also be called a hybrid video encoder or a video encoder.

[0109] The residual calculation unit 204, transform processing unit 206, quantization unit 208, and mode selection unit 260 constitute the forward signal path of the encoder 20; the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, buffer 216, loop filter 220, decoded picture buffer (DPB) 230, inter-frame prediction unit 244, and intra-frame prediction unit 254 constitute the reverse signal path of the video encoder 20, wherein the reverse signal path of the video encoder 20 corresponds to the decoder (see [link to decoder]). Figure 3The signal path of the decoder 30. The inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded image buffer 230, inter-frame prediction unit 244 and intra-frame prediction unit 254 also constitute the "built-in decoder" of the video encoder 20.

[0110] Image and image partitioning (images and blocks)

[0111] Encoder 20 can be used to receive image 17 (or image data 17) via input terminal 201, for example, images in an image sequence forming a video or video sequence. The received image or image data can also be a pre-processed image 19 (or pre-processed image data 19). For simplicity, it is referred to as image 17 in the following description. Image block 17 can also be referred to as the current image or the image to be decoded (especially in video decoding, to distinguish the current image from other images, such as images previously encoded and / or decoded in the same video sequence (i.e., the video sequence that also includes the current image).

[0112] A (digital) image is, or can be viewed as, a two-dimensional array or matrix of samples with intensity values. Samples in the array can also be called pixels (short for pixels). The number of samples in the horizontal and vertical directions (or axes) of the array or image defines the size and / or resolution of the image. Color is typically represented using three color components; that is, an image can be represented by three sample arrays, or an image may include three sample arrays. In RBG format or color space, an image includes corresponding red, green, and blue sample arrays. However, in video decoding, each pixel is typically represented in a luminance and chrominance format or color space, such as YCbCr, which includes a luminance component represented by Y (sometimes replaced by L) and two chrominance components represented by Cb and Cr. The luminance component Y represents luminance or grayscale intensity (e.g., in a grayscale image), while the two chrominance components Cb and Cr represent chrominance or color information components. Therefore, an image in YCbCr format includes a luminance sample array (Y) of luminance sample values ​​and two chrominance sample arrays (Cb and Cr) of chrominance values. An RGB format image can be converted or transformed to YCbCr format and vice versa; this process is also called color transformation or conversion. If the image is monochrome, it can consist only of a luminance sample array. Therefore, an image can be, for example, a luminance sample array in monochrome format or a luminance sample array and two corresponding chrominance sample arrays in 4:2:0, 4:2:2, and 4:4:4 color formats.

[0113] In an embodiment of the video encoder 20, the video encoder 20 may include an image segmentation unit ( Figure 2(Not shown in the image) is used to divide image 17 into multiple (typically non-overlapping) image blocks 203. These blocks may also be called root blocks or macroblocks (H.264 / AVC standard), coding tree blocks (CTB), or coding tree units (CTU) (H.265 / HEVC and VVC standards). Image partitioning units can be used to apply the same block size to all images in a video sequence and the corresponding raster with a defined block size, or to vary the block size between images, subsets, or groups of images, dividing each image into corresponding blocks.

[0114] In other embodiments, the video encoder may be used to directly receive blocks 203 of image 17, for example, one, several, or all of the blocks that make up image 17. Image block 203 may also be referred to as the current image block or the image block to be decoded.

[0115] Although the size of image patch 203 is smaller than that of image 17, like image 17, patch 203 is, or can be considered, a two-dimensional array or matrix of samples with intensity values ​​(sample values). In other words, patch 203 may include, for example, a sample array (e.g., a luminance array in the case of black and white image 17), three sample arrays (e.g., a luminance array and two chrominance arrays in the case of color image 17), or any other number and / or type of array, depending on the applied color format. The number of samples in the horizontal and vertical directions (or axes) of patch 203 defines the size of patch 203. Therefore, a patch can be an M×N (M columns × N rows) sample array, or an M×N transform coefficient array, etc.

[0116] exist Figure 2 In the embodiment of the video encoder 20 shown, the video encoder 20 can be used to encode the image 17 block by block, for example, to perform encoding and prediction for each block 203.

[0117] exist Figure 2 In the embodiment of the video encoder 20 shown, the video encoder 20 can also be used to divide and / or encode an image using slices (also known as video slices), wherein the image can be divided or encoded using one or more slices (typically non-overlapping slices), each slice may include one or more blocks (such as CTUs) or one or more groups of blocks (such as tiles in the H.265 / HEVC and VVC standards or bricks in the VVC standard).

[0118] exist Figure 2In the embodiment of the video encoder 20 shown, the video encoder 20 can also be used to divide and / or encode an image using strip / block groups (also known as video block groups) and / or blocks (also known as video blocks), wherein an image can be encoded using one or more strip / block groups (typically non-overlapping) or an image can be divided into one or more strip / block groups, and each strip / block group can include one or more blocks (e.g., CTUs) or one or more blocks, wherein each block can be rectangular and can include one or more blocks (e.g., CTUs), such as all or part of the blocks.

[0119] Residual calculation

[0120] The residual calculation unit 204 can be used to calculate the residual block 205 (also called residual 205) based on the image block 203 and the prediction block 265 (the prediction block 265 will be described in detail later). For example, the sample value of the prediction block 265 is subtracted from the sample value of the image block 203 on a sample-by-sample (pixel-by-pixel) basis to obtain the residual block 205 in the sample domain.

[0121] Transformation

[0122] The transform processing unit 206 can be used to transform the sample values ​​of the residual block 205, such as by discrete cosine transform (DCT) or discrete sine transform (DST), to obtain transform coefficients 207 in the transform domain. The transform coefficients 207 can also be called transform residual coefficients and represent the residual block 205 in the transform domain.

[0123] Transform processing unit 206 can be used to apply integer approximations of DCT / DST, such as those specified for H.265 / HEVC. These integer approximations are typically scaled by a certain factor compared to orthogonal DCT transforms. Other scaling factors are used as part of the transform process to maintain the norm of the residual block after forward and inverse transforms. Scaling factors are typically selected based on certain constraints, such as scaling factors being powers of 2 (e.g., the bit depth of the transform coefficients), trade-offs between precision and implementation cost, etc. For example, a specific scaling factor for the inverse transform is specified by inverse transform processing unit 212 (and a scaling factor is specified for the corresponding inverse transform in video decoder 30 via inverse transform processing unit 312, etc.), and a corresponding scaling factor for the forward transform can be specified in encoder 20 via transform processing unit 206, etc.

[0124] In an embodiment of the video encoder 20, for example, the video encoder 20 (correspondingly, the transform processing unit 206) can be used to directly output or encode or compress one or more transform parameters of one transform type, so that the video decoder 30 can receive and use the transform parameters for decoding.

[0125] Quantification

[0126] Quantization unit 208 can be used to quantize the transform coefficients 207 (e.g., perform scalar quantization or vector quantization) to obtain quantization coefficients 209. Quantization coefficients 209 can also be referred to as quantization transform coefficients 209 or quantization residual coefficients 209.

[0127] The quantization process can reduce the bit depth associated with some or all of the transform coefficients 207. For example, n-bit transform coefficients can be rounded down to m-bit transform coefficients during quantization, where n is greater than m. The degree of quantization can be modified by adjusting the quantization parameter (QP). For example, for scalar quantization, different scaling can be used to achieve finer or coarser quantization. The smaller the quantization step size, the finer the quantization; the larger the quantization step size, the coarser the quantization. The appropriate quantization step size can be indicated by the quantization parameter (QP). For example, the quantization parameter can be an index applicable to a predefined set of appropriate quantization step sizes. For example, a small quantization parameter can correspond to fine quantization (small quantization step size), while a large quantization parameter can correspond to coarse quantization (large quantization step size), and vice versa. Quantization operations can include division by the quantization step size, while corresponding dequantization or dedequantization operations performed by the dequantization unit 210, etc., can include multiplication by the quantization step size. According to some standards (e.g., HEVC), the quantization parameter can be used in embodiments to determine the quantization step size. Typically, the quantization step size can be calculated from the quantization parameters using a fixed-point approximation of the equations involving division. Additional scaling factors can be introduced into quantization and dequantization to recover the norm of the residual block, as this norm can be modified because scaling is used in the fixed-point approximation of the equations for the quantization step size and quantization parameters. In one exemplary implementation, scaling in the inverse transform and dequantization can be combined. Alternatively, a custom quantization table can be used, and this custom quantization table can be sent from the encoder to the decoder, for example, in the bitstream. Quantization is a lossy operation, and the loss increases with the quantization step size.

[0128] In an embodiment of the video encoder 20, the video encoder 20 (correspondingly, the quantization unit 208) can be used, for example, to directly output or output a quantization parameter (QP) after being encoded by the entropy encoding unit 270, so that the video decoder 30 can receive and use the quantization parameter for decoding.

[0129] Inverse Quantization

[0130] The dequantization unit 210 is used to perform dequantization on the quantization coefficients performed by the quantization unit 208 to obtain the dequantization coefficients 211. For example, it performs a dequantization scheme based on or using the same quantization step size as the quantization unit 208 to perform the quantization scheme performed by the quantization unit 208. The dequantization coefficients 211 can also be called dequantization residual coefficients 211, which correspond to the transform coefficients 207. However, due to the loss caused by quantization, the dequantization coefficients 211 are usually not exactly the same as the transform coefficients.

[0131] Inverse Transformation

[0132] The inverse transform processing unit 212 is used to perform the inverse transform of the transform performed by the transform processing unit 206, such as the inverse discrete cosine transform (DCT) or the inverse discrete sine transform (DST), to obtain the reconstructed residual block 213 (or the corresponding dequantization coefficients 213) in the sample domain. The reconstructed residual block 213 can also be referred to as the transform block 213.

[0133] reconstruction

[0134] The reconstruction unit 214 (e.g., adder or summer 214) is used to add the transform block 213 (i.e., the reconstruction residual block 213) to the prediction block 265, for example, by adding the sample values ​​of the reconstruction residual block 213 and the sample values ​​of the prediction block 265 sample by sample to obtain the reconstruction block 215 in the sample domain.

[0135] Filtering

[0136] Loop filter unit 220 (or simply "loop filter" 220) is used to filter reconstructed block 215 to obtain filtered block 221, or typically to filter reconstructed samples to obtain filtered sample values. For example, the loop filter unit is used to smooth pixel abrupt changes or improve video quality. Loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. For example, loop filter unit 220 may include a deblocking filter, a SAO filter, and an ALF filter; the filtering process may be deblocking filtering, SAO filtering, and ALF filtering in that order. In another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop shaper) is added. This process is performed before deblocking filtering. In another example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra-sub-partition (ISP) edges. Although in Figure 2 The middle loop filter unit 220 is shown as an in-loop filter, but in other configurations, the loop filter unit 220 can be implemented as a rear loop filter. Filter block 221 can also be referred to as filter reconstruction block 221.

[0137] In an embodiment of the video encoder 20, the video encoder 20 (correspondingly, the loop filter unit 220) can be used to directly output or encode loop filter parameters (e.g., SAO filter parameters, ALF filter parameters, or LMCS parameters) through the entropy coding unit 270, so that the decoder 30 can receive and use the same or different loop filter parameters for decoding.

[0138] Decoding image buffer

[0139] The decoded picture buffer (DPB) 230 can be a memory that stores a reference picture or typically stores reference picture data, for use when the video encoder 20 encodes video data. The DPB 230 can be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM) (including synchronous DRAM (SDRAM)), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The decoded picture buffer (DPB) 230 can be used to store one or more filtered blocks 221. The decoded picture buffer 230 can also be used to store other previously filtered blocks (e.g., previously reconstructed and filtered blocks 221) of the same current picture or different pictures (e.g., previously reconstructed pictures), and can provide complete previously reconstructed (i.e., decoded) pictures (and corresponding reference blocks and samples) and / or partially reconstructed current pictures (and corresponding reference blocks and samples), for example, for inter-frame prediction. The decoded image buffer 230 can also be used to store one or more unfiltered reconstruction blocks 215, or typically store unfiltered reconstruction samples, such as reconstruction blocks 215 that have not been filtered by the loop filter unit 220, or reconstruction blocks or reconstruction samples that have not undergone any other processing.

[0140] Pattern selection (partitioning and prediction)

[0141] The mode selection unit 260 includes a partitioning unit 262, an inter-frame prediction unit 244, and an intra-frame prediction unit 254. It is used to receive or obtain raw image data, such as raw block 203 (current block 203 of the current image 17), and reconstructed image data, such as filtered and / or unfiltered reconstructed samples or blocks of the same (current) image and / or one or more previously decoded images, from the decoded image buffer 230 or other buffers (e.g., a line buffer, not shown in the figure). The reconstructed image data is used as reference image data for predictions such as inter-frame prediction or intra-frame prediction to obtain prediction block 265 or prediction value 265.

[0142] The mode selection unit 260 can be used to determine or select a partitioning mode for the current block prediction mode (including no partitioning mode) and prediction mode (e.g., intra-frame or inter-frame prediction mode), and generate a corresponding prediction block 265, which is used for the calculation of residual block 205 and the reconstruction of reconstruction block 215.

[0143] In embodiments of the mode selection unit 260, the mode selection unit 260 can be used to select partitioning and prediction modes (e.g., from modes supported or available by the mode selection unit 260). The partitioning and prediction modes provide an optimal match, i.e., minimum residual (minimum residual is more favorable for compression for transmission or storage), or minimum indication overhead (minimum indication overhead is more favorable for compression for transmission or storage), or a balance between the two. The mode selection unit 260 can be used to determine the partitioning and prediction modes based on rate distortion optimization (RDO), i.e., selecting the prediction mode that provides minimum rate distortion. In this document, terms such as "optimal," "minimum," and "most suitable" do not necessarily refer to "best," "minimum," or "most suitable" overall; they can also refer to situations that meet termination or selection criteria. For example, a value exceeding or falling below a threshold or other limitation may lead to a "suboptimal choice," but will reduce complexity and processing time.

[0144] In other words, partitioning unit 262 can be used, for example, to partition images of a video sequence into a series of coding tree units (CTUs) by iteratively using quad-tree partitioning (QT), binary partitioning (BT), or triple-tree partitioning (TT), or any combination thereof, and to further partition CTUs 203 into smaller block portions or sub-blocks (which again form blocks), and partitioning unit 262 can be used, for example, to make predictions for each of the block portions or sub-blocks, wherein mode selection includes selecting the tree structure of the partitioned blocks 203 and the prediction mode applied to each block portion or sub-block.

[0145] The following describes in detail the partitioning (e.g., partitioning unit 260) and prediction processing (e.g., inter-frame prediction unit 244 and intra-frame prediction unit 254) performed by the video encoder 20.

[0146] Divide into modules

[0147] The partitioning unit 262 can be used to partition images in a video sequence into a sequence of coding tree units (CTUs). The partitioning unit 262 can divide (or subdivide) the coding tree unit (CTU) 203 into smaller parts, such as smaller square or rectangular blocks. For an image with three sample arrays, a CTU consists of an N×N luma sample block and two corresponding chroma sample blocks. In the developing universal video coding (VVC), the maximum allowed size of the luma block in the CTU is specified as 128×128, but in the future, a value different from 128×128 can be specified, such as 256×256. Image CTUs can be clustered / grouped into slices / tile groups, tiles, or bricks. A tile covers a rectangular area of ​​the image and can be divided into one or more bricks. A brick consists of multiple rows of CTUs within a tile. A tile that is not divided into multiple bricks can be called a brick. However, a brick is a proper subset of a tile and is not called a tile. VVC supports two tile grouping modes: raster scan strip / tile grouping mode and rectangular strip mode. In raster scan tile grouping mode, a strip / tile group consists of a sequence of tiles in a tiled raster scan of an image. In rectangular strip mode, a strip consists of multiple tiles of an image that together form a rectangular region of the image. The tiles within a rectangular strip are arranged in the raster scan order of the strip's tiles. These smaller tiles (also called sub-tiles) can be further subdivided into smaller parts. This is also called tree partitioning or hierarchical tree partitioning, where the root tile at root level 0 (level 0, depth 0), etc., can be recursively partitioned into two or more tiles at the next lower tree level, such as nodes at tree level 1 (level 1, depth 1). These tiles can then be further partitioned into two or more tiles at the next lower level, such as tree level 2 (level 2, depth 2), etc., until the partitioning ends due to the satisfaction of a termination criterion (e.g., reaching the maximum tree depth or minimum tile size). Tiles that are not further partitioned are also called leaf tiles or leaf nodes of the tree. A tree divided into two parts is called a binary tree (BT), a tree divided into three parts is called a ternary tree (TT), and a tree divided into four parts is called a quad tree (QT).

[0148] For example, a coding tree unit (CTU) can be or includes: a CTB of luminance samples, two corresponding CTBs of chroma samples of an image with three sample arrays, a CTB of samples of a monochrome image or an image decoded using three separate color planes and a syntax structure for decoding the samples. Correspondingly, a coding tree block (CTB) can be an N×N sample block, where N can be set to a value to divide the components into CTBs; this is one type of partitioning. A coding unit (CU) can be or includes: a coding block of luminance samples, two corresponding coding blocks of chroma samples of an image with three sample arrays, a coding block of samples of a monochrome image or an image decoded using three separate color planes and a syntax structure for decoding the samples. Correspondingly, a coding block (CB) can be an M×N sample block, where M and N can be set to certain values ​​to divide the CTB into coding blocks; this is one type of partitioning.

[0149] In an embodiment, for example, according to HEVC, a coding tree unit (CTU) can be partitioned into CUs using a quadtree structure represented as a coding tree. At the leaf CU level, it is determined whether the image region is decoded via inter-frame (temporal) prediction or intra-frame (spatial) prediction. Each leaf CU can be further partitioned into one, two, or four PUs depending on the PU partitioning type. Within a PU, the same prediction process is performed, and relevant information is sent to the decoder on a PU-by-PU basis. After obtaining the residual block through the prediction process, the leaf CU can be partitioned into transform units (TUs) according to other quadtree structures similar to the coding tree used for the CUs, depending on the PU partitioning type.

[0150] For example, in one embodiment, based on the latest video decoding standard currently under development (called Versatile Video Coding (VVC)), a combined quadtree of nested multi-type trees (e.g., binary and ternary trees) is used to partition, for example, the segmented structure used to divide coding tree units. Within the coding tree structure of a coding tree unit, a CU can be square or rectangular. For example, a coding tree unit (CTU) is first partitioned using a quadtree. Then, the leaf nodes of the quadtree can be further partitioned using a multi-type tree structure. There are four partitioning types in the multi-type tree structure: vertical binary tree partitioning (SPLIT_BT_VER), horizontal binary tree partitioning (SPLIT_BT_HOR), vertical ternary tree partitioning (SPLIT_TT_VER), and horizontal ternary tree partitioning (SPLIT_TT_HOR). The leaf nodes of the multi-type tree are called coding units (CUs). This segment is used for prediction and transform processing and does not require further partitioning; further partitioning is only needed when the maximum transform length of the CU is too large. This means that, in most cases, CU, PU, ​​and TU have the same block size in a quadtree with a nested multi-type tree coding block structure. An anomaly occurs when the maximum supported transform length is less than the width or height of the color component of the CU. VVC establishes a unique indication mechanism for partitioning information in a quadtree with nested multi-type tree decoding. In this mechanism, the coding tree unit (CTU) serves as the root of the quadtree and is first partitioned through the quadtree structure. Then, each quadtree leaf node is further partitioned through a multi-type tree structure (when it is too large to be further partitioned). In the multi-type tree structure, a first flag (mtt_split_cu_flag) indicates whether a node should be further partitioned; when further partitioning is needed, a second flag (mtt_split_cu_vertical_flag) indicates the partition direction, and then a third flag (mtt_split_cu_binary_flag) indicates whether the partition is a binary or ternary partition. Based on the values ​​of `mtt_split_cu_vertical_flag` and `mtt_split_cu_binary_flag`, the decoder can deduce the multi-type tree partitioning mode (MttSplitMode) for the CU according to a predefined rule or a table. It should be noted that for certain designs, such as the 64×64 luma block and 32×32 chroma block pipeline design in the VVC hardware decoder, TT partitioning is prohibited when the width or height of the luma encoding block is greater than 64. Figure 6As shown. TT partitioning is also prohibited when the width or height of the chroma coding block is greater than 32. Pipeline design divides the image into virtual pipeline data units (VPDUs), which are defined as non-overlapping units in the image. In a hardware decoder, consecutive VPDUs are processed simultaneously by multiple pipeline stages. In most pipeline stages, the VPDU size is roughly proportional to the buffer size, so keeping the VPDU size small is important. In most hardware decoders, the VPDU size can be set to the maximum transform block (TB) size. However, in VVC, ternary tree (TT) and binary tree (BT) partitioning can lead to larger VPDU sizes.

[0151] Additionally, it should be noted that when a portion of a tree node block extends beyond the bottom or right boundary of the image, the tree node block is forcibly divided until all samples of each decoded CU are within the image boundary.

[0152] For example, depending on the block size, the intra sub-partition (ISP) tool can divide the lumen intra-prediction block vertically or horizontally into 2 or 4 sub-parts.

[0153] In one example, the mode selection unit 260 of the video encoder 20 can be used to perform any combination of the partitioning techniques described herein.

[0154] As described above, the video encoder 20 is used to determine or select the best or optimal prediction mode from a set (e.g., predetermined) of prediction modes. This set of prediction modes may include, for example, intra-frame prediction modes and / or inter-frame prediction modes.

[0155] Intra-frame prediction

[0156] This set of intra-prediction modes can include 35 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes defined in HEVC, or it can include 67 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes defined in VVC, etc. For example, several traditional angular intra-prediction modes are adaptively replaced with wide-angle intra-prediction modes for non-square blocks as defined in VVC. As another example, to avoid the division operation of DC prediction, only the longer side is used to calculate the average value of non-square blocks. Moreover, the intra-prediction results of the planar mode can be modified using the position-dependent intra-prediction combination (PDPC) method.

[0157] Intra-prediction unit 254 is used to generate intra-prediction block 265 using reconstructed samples of neighboring blocks of the same current image based on intra-prediction modes in the intra-prediction mode set.

[0158] Intra-prediction unit 254 (or typically mode selection unit 260) is also used to output intra-prediction parameters (or typically information on the selected intra-prediction mode of the block) to entropy coding unit 270 in the form of syntax element 266, so as to include the intra-prediction parameters in the encoded image data 21, so that the video decoder 30 can, for example, receive and use the prediction parameters for decoding.

[0159] Inter-frame prediction

[0160] The set (or possible) inter-frame prediction modes depend on the available reference image (i.e., a previously at least partially decoded image stored in the DBP 230) and other inter-frame prediction parameters, such as whether the entire reference image or only a portion of the reference image (e.g., the search window region around the current block) is used to search for the best-matching reference block, and / or whether pixel interpolation is applied, such as half-pixel, quarter-pixel, and / or 1 / 16-pixel interpolation, or no pixel interpolation is applied.

[0161] In addition to the prediction modes mentioned above, skip mode, direct mode and / or other inter-frame prediction modes can also be applied.

[0162] For example, in extended fusion prediction, the fusion candidate list consists of five candidate types in sequence: spatial MVP of spatially adjacent CUs, temporal MVP of juxtaposed CUs, history-based MVP of a FIFO table, paired average MVP, and zero MV. Decoder-side motion vector refinement (DMVR) based on bidirectional matching can be used to increase the accuracy of MV in fusion mode. Merge mode with MVD (MMVD) is a fusion mode with motion vector difference. The MMVD flag is indicated immediately after sending the skip flag and fusion flag to indicate whether MMVD mode is used for the CU. An adaptive motion vector resolution (AMVR) scheme at the CU level can be used. The AMVR scheme supports decoding the MVD of the CU with different accuracies. The MVD of the current CU is adaptively selected according to the prediction mode of the current CU. When the CU is decoding in fusion mode, a combined inter / intra prediction (CIIP) mode can be applied to the current CU. The inter-frame prediction signal and the intra-frame prediction signal are weighted and averaged to obtain the CIIP prediction. In affine motion compensation prediction, the affine motion field of the block is described by the motion information of motion vectors from two control points (4-parameters) or three control points (6-parameters). Subblock-based temporal motion vector prediction (SbTMVP) is similar to temporal motion vector prediction (TMVP) in HEVC, but it predicts the motion vectors of sub-CUs in the current CU. Bidirectional optical flow (BDOF), formerly known as BIO, is a simpler version that requires less computation, especially in terms of the number of multiplications and multiplier size. In the triangular partitioning mode, the CU is uniformly divided into two triangular parts using both diagonal and anti-diagonal partitioning. Furthermore, the bidirectional prediction mode extends the simple averaging to allow weighted averaging of the two prediction signals.

[0163] Inter-frame prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (neither of which is in the same frame). Figure 2(As shown in the figure). The motion estimation unit can be used to receive or acquire image block 203 (current image block 203 of current image 17) and decoded image 231, or at least one or more previously reconstructed blocks (e.g., reconstructed blocks of one or more other / different previously decoded images 231) for motion estimation. For example, the video sequence may include the current image and the previously decoded image 231, or, in other words, the current image and the previously decoded image 231 may be part of or constitute the image sequence that makes up the video sequence.

[0164] For example, encoder 20 can be used to select a reference block from multiple reference blocks of the same or different images from multiple other images and provide the offset (spatial offset) between the position (x-coordinate, y-coordinate) of the reference image (or reference image index) and / or the position of the reference block and the position of the current block as an inter-frame prediction parameter of the motion estimation unit. This offset is also called the motion vector (MV).

[0165] The motion compensation unit is used to acquire (e.g., receive) inter-frame prediction parameters and perform inter-frame prediction based on or through the inter-frame prediction parameters to obtain inter-frame prediction block 265. Motion compensation performed by the motion compensation unit may include extracting or generating prediction blocks based on motion / block vectors determined by motion estimation (possibly performing interpolation to improve sub-pixel accuracy). Interpolation filtering can generate other pixel samples from known pixel samples, potentially increasing the number of candidate prediction blocks that can be used to decode image blocks. Upon receiving the motion vector of the PU for the current image block, the motion compensation unit can locate the prediction block that the motion vector points to in a list of reference images.

[0166] The motion compensation unit can also generate syntax elements associated with blocks and video stripes, which are used by the video decoder 30 to decode image blocks of the video stripes. In addition to, or as a substitute for, stripes and their corresponding syntax elements, chunk groups and / or chunks, along with their corresponding syntax elements, can be generated or used.

[0167] Entropy decoding

[0168] For example, the entropy coding unit 270 is used to apply an entropy coding algorithm or scheme (e.g., variable-length coding (VLC) scheme, context-adaptive VLC (CALVC) scheme, arithmetic decoding scheme, binarization algorithm, context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) decoding, or other entropy coding methods or techniques) or a bypass (uncompressed) scheme to quantization coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, loop filtering parameters, and / or other syntax elements to obtain encoded image data 21 that can be output by the output unit 272 in the form of an encoded bitstream 21, etc., so that the video decoder 30 can receive and use these parameters for decoding. The encoded bitstream 21 can be sent to the video decoder 30 or stored in memory for aggregation so that it can be subsequently sent or retrieved by the video decoder 30.

[0169] Other structural variations of the video encoder 20 can be used to encode video streams. For example, for certain blocks or frames, the non-transform-based encoder 20 can directly quantize the residual signal without the transform processing unit 206. In another implementation, the encoder 20 can combine the quantization unit 208 and the inverse quantization unit 210 into a single unit.

[0170] Decoder and Decoding Method

[0171] Figure 3 Example of a video decoder 30 used to implement the techniques of this application. The video decoder 30 is used to receive encoded image data 21 (e.g., encoded bitstream 21) encoded by encoder 20, for example, to obtain a decoded image 331. The encoded image data or bitstream includes information for decoding the encoded image data, such as data (and / or chunks or blocks) representing image blocks of encoded video stripes and associated syntax elements.

[0172] exist Figure 3In the example, decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), a loop filter 320, a decoded picture buffer (DBP) 330, a mode application unit 360, an inter-frame prediction unit 344, and an intra-frame prediction unit 354. The inter-frame prediction unit 344 may be or include a motion compensation unit. In some examples, video decoder 30 may perform... Figure 2 The encoding channels described in the video encoder 100 are largely inverse decoding channels.

[0173] As explained regarding encoder 20, the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded picture buffer (DPB) 230, inter-frame prediction unit 344, and intra-frame prediction unit 354 also constitute the "built-in decoder" of video encoder 20. Therefore, inverse quantization unit 310 can have the same function as inverse quantization unit 110, inverse transform processing unit 312 can have the same function as inverse transform processing unit 212, reconstruction unit 314 can have the same function as reconstruction unit 214, loop filter 320 can have the same function as loop filter 220, and decoded picture buffer 330 can have the same function as decoded picture buffer 230. Therefore, the explanation of the corresponding units and functions of video encoder 20 is correspondingly applicable to the corresponding units and functions of video decoder 30.

[0174] Entropy Decoding

[0175] The entropy decoding unit 304 is used to parse the bitstream 21 (or commonly referred to as coded image data 21) and perform entropy decoding on the coded image data 21 to obtain the quantization coefficients 309 and / or decoding parameters. Figure 3 (Not shown in the image) , such as any or all of the following: inter-frame prediction parameters (e.g., reference image index and motion vector), intra-frame prediction parameters (e.g., intra-frame prediction mode or index), transform parameters, quantization parameters, loop filtering parameters, and / or other syntax elements. The entropy decoding unit 304 can be used to perform a decoding algorithm or scheme corresponding to the encoding scheme described by the entropy encoding unit 270 of the encoder 20. The entropy decoding unit 304 can also be used to provide inter-frame prediction parameters, intra-frame prediction parameters, and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units of the decoder 30. The video decoder 30 can receive syntax elements at the video stripe level and / or video block level. In addition to or as a substitute for stripes and corresponding syntax elements, it can receive and / or use chunk groups and / or chunks and corresponding syntax elements.

[0176] Inverse Quantization

[0177] The dequantization unit 310 can be used to receive quantization parameters (QP) (or generally information related to dequantization) and quantization coefficients from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304, etc.), and dequantize the decoded quantization coefficients 309 based on the quantization parameters to obtain dequantization coefficients 311, which may also be referred to as transform coefficients 311. The dequantization process may include: using the quantization parameters of each video block in the video strip (or block or group of blocks) determined by the video encoder 20 to determine the degree of quantization to be applied and the degree of dequantization.

[0178] Inverse Transformation

[0179] The inverse transform processing unit 312 can be used to receive the dequantized coefficients 311 (also referred to as transform coefficients 311) and transform the dequantized coefficients 311 to obtain the reconstructed residual block 213 in the sample domain. The reconstructed residual block 213 can also be referred to as transform block 213. The transform can be an inverse transform, such as inverse DCT, inverse DST, inverse integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 can also be used to receive transform parameters or corresponding information from the encoded image data 21 (e.g., through parsing and / or decoding, e.g., through the entropy decoding unit 304) to determine the transform to be applied to the dequantized coefficients 311.

[0180] reconstruction

[0181] Reconstruction unit 314 (e.g., adder or summer 314) can be used to add reconstruction residual block 313 to prediction block 365 to obtain reconstruction block 315 in the sample domain, for example, by adding the sample values ​​of reconstruction residual block 313 to the sample values ​​of prediction block 365.

[0182] Filtering

[0183] Loop filtering unit 320 (in or after the decoding loop) is used to filter the reconstructed block 315 to obtain filtered block 321, smoothing pixel abrupt changes or improving video quality. Loop filtering unit 320 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. For example, loop filtering unit 320 may include a deblocking filter, a SAO filter, and an ALF filter; the filtering sequence may be deblocking filtering, SAO filtering, and ALF filtering. In another example, a process called luma mapping with chroma scaling (LMCS) (i.e., an adaptive in-loop shaper) is added. This process is performed before deblocking filtering. In another example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra-sub-partition (ISP) edges. Although in Figure 3 The middle loop filter unit 320 is shown as an in-loop filter, but in other configurations, the loop filter unit 320 can be implemented as a rear loop filter.

[0184] Decoding image buffer

[0185] Next, the decoded video block 321 of an image is stored in the decoded image buffer 330. The decoded image 331 is stored in the decoded image buffer 330 as a reference image for subsequent motion compensation and / or subsequent corresponding output for display of other images.

[0186] The decoder 30 is used, for example, to output a decoded image 311 via output terminal 312 for presentation to a user or for viewing by a user.

[0187] predict

[0188] The function of the inter-frame prediction unit 344 can be the same as that of the inter-frame prediction unit 244 (especially the motion compensation unit); the function of the intra-frame prediction unit 354 can be the same as that of the inter-frame prediction unit 254, and it determines the splitting mode and performs prediction based on the splitting and / or prediction parameters or corresponding information received from the coded image data 21 (e.g., parsing and / or decoding by the entropy decoding unit 304, etc.). The mode application unit 360 can be used to perform block-by-block prediction (intra-frame prediction or inter-frame prediction) based on the reconstructed image, reconstructed blocks, or corresponding samples (filtered or unfiltered) to obtain prediction blocks 365.

[0189] When the video stripe is decoded as an intra-decoded (I) stripe, the intra-prediction unit 354 of the mode application unit 360 is used to generate a prediction block 365 for the image block of the current video stripe based on the indicated intra-prediction mode and data from previously decoded blocks of the current image. When the video image is decoded as an inter-decoded (i.e., B or P) stripe, the inter-prediction unit 344 (e.g., motion compensation unit) of the mode application unit 360 is used to generate a prediction block 365 for the video block of the current video stripe based on motion vectors and other syntax elements received from the entropy decoding unit 304. For inter-prediction, the prediction block can be generated from a reference image in a list of reference images. The video decoder 30 can construct the reference frame list, list 0 and list 1, based on the reference images stored in the DPB 330 using a default construction technique. In addition to embodiments using stripes (e.g., video stripes), the same or similar processes can be applied to embodiments using chunk groups (e.g., video chunk groups) and / or chunks (e.g., video chunks), for example, video can be decoded using I, P, or B chunk groups and / or chunks.

[0190] The pattern application unit 360 is used to determine the prediction information of video blocks in the current video strip by parsing motion vectors or related information and other syntax elements, and to use the prediction information to generate prediction blocks for the current video block being decoded. For example, the pattern application unit 360 uses some received syntax elements to determine the prediction mode (e.g., intra-frame or inter-frame prediction) for decoding video blocks in the video strip, the inter-frame prediction stripe type (e.g., B stripe, P stripe, or GPB stripe), the construction information of one or more of the reference image list of the stripe, the motion vector of each inter-frame coded video block of the stripe, the inter-frame prediction state of each inter-frame decoded video block of the stripe, and other information to decode video blocks in the current video stripe. In addition to embodiments using stripes (e.g., video stripes), the same or similar process can be applied to embodiments using chunk groups (e.g., video chunk groups) and / or chunks (e.g., video chunks), for example, video can be decoded using I, P, or B chunk groups and / or chunks.

[0191] exist Figure 3 In the embodiment of the video decoder 30 shown, the video decoder 30 can be used to divide and / or decode an image using slices (also known as video slices), wherein the image can be divided or decoded using one or more slices (typically non-overlapping slices), each slice may include one or more blocks (such as CTUs) or one or more groups of blocks (such as tiles in the H.265 / HEVC and VVC standards or bricks in the VVC standard).

[0192] exist Figure 3 In the embodiment of the video decoder 30 shown, the video decoder 30 can be used to divide and / or decode an image using strip / block groups (also known as video block groups) and / or blocks (also known as video blocks), wherein an image can be decoded using one or more strip / block groups (typically non-overlapping) or an image can be divided into one or more strip / block groups, and each strip / block group can include one or more blocks (e.g., CTUs) or one or more blocks, wherein each block can be rectangular and can include one or more blocks (e.g., CTUs), such as all or part of the blocks.

[0193] Other variations of the video decoder 30 can be used to decode encoded image data 21. For example, the decoder 30 can generate an output video stream without the loop filtering unit 320. For example, for certain blocks or frames, the non-transform-based decoder 30 can directly quantize the residual signal without the inverse transform processing unit 312. In another implementation, the video decoder 30 can combine the inverse quantization unit 310 and the inverse transform processing unit 312 into a single unit.

[0194] It should be understood that in encoder 20 and decoder 30, the processing results of the current stage can be further processed and then output to the next stage. For example, after interpolation filtering, motion vector derivation, or loop filtering, the processing results of interpolation filtering, motion vector derivation, or loop filtering can be further modified (clipped) or shifted (shifted).

[0195] It should be noted that further operations can be performed on the derived motion vector of the current block (including but not limited to control point motion vectors in affine mode, affine, planar, sub-block motion vectors in ATMVP mode, temporal motion vectors, etc.). For example, the value of the motion vector can be restricted to a predefined range based on its representation bit. If the representation bit of the motion vector is bitDepth, the value range of the motion vector is –2^(bitDepth–1) to 2^(bitDepth–1)–1, where the “^” sign represents exponentiation. For example, if bitDepth is set to 16, the value range is –32768 to 32767; if bitDepth is set to 18, the value range is –131072 to 131071. For example, the value of the derived motion vector (e.g., the MV of four 4×4 sub-blocks in an 8×8 block) can be restricted so that the maximum difference between the integer parts of the MV of the four 4×4 sub-blocks does not exceed N pixels, for example, not more than 1 pixel. This section provides two methods for limiting motion vectors based on bitDepth.

[0196] Figure 4 This is a schematic diagram of a video decoding device 400 provided according to one embodiment of the present invention. As described herein, the video decoding device 400 is suitable for implementing the disclosed embodiments. In one embodiment, the video decoding device 400 may be a decoder (e.g., Figure 1A The video decoder 30 or encoder (e.g., in the video decoder 30) or encoder (e.g., Figure 1A (Video encoder 20 in the middle).

[0197] The video decoding device 400 includes: an input port 410 and a receiving unit (Rx) 420 for receiving data; a processor, logic unit, or central processing unit (CPU) 430 for processing data; a transmitter unit (Tx) 440 and an output port 450 for transmitting data; and a memory 460 for storing data. The video decoding device 400 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the input port 410, the receiver unit 420, the transmitter unit 440, and the output port 450 for the input or output of optical or electrical signals.

[0198] Processor 430 is implemented in both hardware and software. Processor 430 can be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. Processor 430 communicates with ingress port 410, receiver unit 420, transmitter unit 440, egress port 450, and memory 460. Processor 430 includes a decoding module 470. Decoding module 470 implements the embodiments disclosed above. For example, decoding module 470 implements, processes, prepares, or provides various decoding operations. Therefore, including decoding module 470 significantly enhances the functionality of video decoding device 400 and affects the transitions of video decoding device 400 to different states. Alternatively, decoding module 470 can be implemented as instructions stored in memory 460 and executed by processor 430.

[0199] Memory 460 may include one or more disks, tape drives, and solid-state drives, and may be used as an overflow data storage device to store programs when such programs are selected for execution, as well as instructions and data read during program execution. For example, memory 460 may be volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0200] Figure 5 A simplified block diagram of a device 500 provided for an exemplary embodiment, which can be used as... Figure 1A One or both of the source device 12 and the destination device 14.

[0201] The processor 502 in device 500 may be a central processing unit. Alternatively, processor 502 may be any other type of device or multiple devices capable of manipulating or processing information that is present or developed in the future. Although the disclosed implementation may be implemented with a single processor (e.g., processor 502), speed and efficiency may be improved by using more than one processor.

[0202] In one implementation, the memory 504 in device 500 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory 504. Memory 504 may include code and data 506 accessed by processor 502 via bus 512. Memory 504 may also include an operating system 508 and an application program 510, which includes at least one program that causes processor 502 to perform the methods described herein. For example, application program 510 may include applications 1 to N, which also include a video decoding application that performs the methods described herein.

[0203] The device 500 may also include one or more output devices, such as a display 518. In one example, the display 518 may be a touch-sensitive display that combines a display with a touch-sensitive element operable to sense touch input. The display 518 may be coupled to the processor 502 via a bus 512.

[0204] Although described herein as a single bus, bus 512 of device 500 may consist of multiple buses. Furthermore, auxiliary memory 514 may be directly coupled to other components of device 500 or accessible via a network, and may comprise a single integrated unit (e.g., a memory card) or multiple units (e.g., multiple memory cards). Therefore, device 500 can be implemented in a variety of configurations.

[0205] Current hybrid video codecs employ predictive decoding. The video sequence is subdivided into pixel blocks, which are then decoded. Instead of decoding pixel-by-pixel, the entire block is predicted using previously encoded pixels that are spatially or temporally close to it. The encoder only further processes the difference between the block and its prediction. This further processing typically involves transforming the block pixels into coefficients in the transform domain. The coefficients can then be further compressed (e.g., through quantization) and further compacted (e.g., through entropy decoding) to form a bitstream. The bitstream may also include any indicative information that enables the decoder to decode the encoded video. For example, indicative information may include settings about the encoder setup, such as the size of the input image, frame rate, quantization step size representation, predictions applied to the image blocks, etc.

[0206] The difference between a block and its prediction is called the block's residual. More specifically, each pixel in a block has a residual, which is the difference between the pixel's intensity level and its predicted intensity level. The intensity level of a pixel is called its pixel value. The residuals of all pixels in a block are collectively referred to as the block's residual. In other words, a block has residuals, and the residuals are the set or matrix of residuals that include all pixels in the block.

[0207] Temporal prediction utilizes the temporal correlation between images (also called frames) in a video. Temporal prediction is also called inter-frame prediction because it uses the correlation between different video frames for prediction. Therefore, the block to be decoded (also called the current block) is predicted from one or more previously decoded images (called reference images). These reference images are not necessarily images preceding the current image in the video sequence's display order. The encoder can encode the images in a decoding order different from the display order. As a prediction of the current block, a juxtaposed block (called a prediction value) in the reference images can be determined. The juxtaposed block can be located in the reference image at the same position as the current block in the current image. This prediction is accurate for stationary image regions (i.e., image regions that have not moved from one image to another).

[0208] In encoders, motion estimation is typically used to obtain motion-compensated predictions. The current block is predicted from the block located at the position represented by the motion vector in the reference image. The motion vector points from the position of the juxtaposed block to the position of the current block (and vice versa, depending on the notation convention). To enable the decoder to determine the prediction of the same current block as the encoder, motion vectors can be indicated in the bitstream. To further reduce the indication overhead of indicating motion vectors for each block, the motion vectors themselves can be estimated. Motion vector estimation can be performed based on the motion vectors of neighboring blocks in the spatial and / or temporal domains of the current block.

[0209] The prediction for the current block can be calculated using a reference image or by weighting predictions obtained from two or more reference images. A reference image can be an adjacent image, i.e., an image immediately preceding or following the current image in the display order, as adjacent images are most likely to be similar to the current image. A reference image can also be any image preceding or following the current image in the display order and preceding the current image in the bitstream (decoding order). This is advantageous, for example, in cases where there is occlusion and / or non-linear motion in the video content. Reference images can be indicated in the bitstream.

[0210] A special mode of inter-frame prediction is bidirectional prediction, in which predictions for the current block are generated using two reference images. Specifically, the two predictions determined separately in the two reference images are combined into the prediction signal for the current block. Bidirectional prediction can predict the current block more accurately than unidirectional prediction (i.e., prediction using only a single reference image). More accurate predictions result in smaller differences between the pixels of the current block and the predictions (i.e., smaller residuals), which allows for more efficient encoding, requiring fewer decoding bits.

[0211] To provide more accurate motion estimation, the resolution of the reference image can be improved, for example, by interpolating samples between pixels. Fractional pixel interpolation can be performed by a weighted average of the nearest pixels. For example, bilinear interpolation can be used in the case of half-pixel resolution. Other fractional pixels can be calculated as the average of the nearest pixels, for example, by weighting the average by the inverse of the distance between the corresponding nearest pixel and the pixel being predicted.

[0212] For example, motion vectors can be estimated by calculating the similarity between the current block and the corresponding predicted block pointed to by the candidate motion vectors in the reference image. To reduce complexity, the number of candidate motion vectors can be reduced by restricting them to a certain search space. For example, the search space can be defined by the number and / or positions of pixels in the reference image that correspond to the position of the current block in the current image. Alternatively, candidate motion vectors can be defined by a list of candidate motion vectors consisting of motion vectors from neighboring blocks.

[0213] Motion vectors are typically determined, at least partially, on the encoding side and pointed to by the decoder within the decoded bitstream. Motion vectors can also be derived in the decoder. In this case, the current block is not available in the decoder and cannot be used to calculate the similarity between the current block and any block in the reference image to which the candidate motion vector points. Therefore, a template can be used instead of the current block, which can be constructed from pixels of previously decoded blocks. For example, previously decoded pixels adjacent to the current block can be used. This motion estimation offers the advantage of reduced indication: motion vectors are derived in the same way in both the encoder and decoder, thus eliminating the need for indication. On the other hand, the accuracy of this motion estimation may be lower.

[0214] To balance accuracy and indication overhead, motion vector estimation can be divided into two steps: motion vector derivation and motion vector refinement. For example, motion vector derivation may include selecting motion vectors from a candidate list. The selected motion vectors may be further refined, for example, by searching within the search space. The search within the search space is based on calculating a cost function for each candidate motion vector, i.e., for each candidate position in the block pointed to by the candidate motion vector.

[0215] Document JVET-D0029: Decoder-Side Motion Vector Correction Based on Bidirectional Template Matching, by X. Chen, J. An, and J. Zheng (available at http: / / phenix.it-sudparis.eu / jvet / site), illustrates motion vector correction in which a first motion vector with integer pixel resolution is found, and further correction is performed by searching the search space around the first motion vector at half-pixel resolution. Here, pixel resolution (e.g., integer or half-integer) describes the resolution of the search space, i.e., the displacement of the searched point relative to the uncorrected motion vector input to the process. Therefore, the search coordinates during the correction phase do not necessarily coincide with the actual pixel coordinates on the image plane.

[0216] Motion vector correction can be performed in the decoder without the assistance of the encoder. The decoder loop in the encoder can use the same correction to obtain the corresponding reference image. The correction can be performed by determining a template, determining the search space, and finding the location of the reference image portion in the search space that best matches the template. The location of the best-matching portion determines the optimal motion vector, which is then used to obtain the prediction for the current block (i.e., the block being reconstructed).

[0217] In one embodiment, such as Figure 6 As shown, inter-frame prediction methods include:

[0218] S601: Get the initial motion vector of the current block.

[0219] An initial motion vector MV0 is obtained, which can be considered a first estimate or approximation of the precise motion vector. For example, MV0 can be selected from a list of candidate motion vectors, which may include motion vectors from at least one block adjacent to the current block. Alternatively, MV0 can be obtained on the encoding side through block matching and indicated to the decoding side within the bitstream. Accordingly, on the decoding side, the initial motion vector MV0 can be obtained from the bitstream. For example, an index of the candidate list is extracted from the bitstream, and the candidate motion vector identified by that index is used as the initial motion vector MV0.

[0220] Alternatively, the coordinates of MV0 can be extracted directly from the bitstream. It should be noted that this application is not limited to any particular method of obtaining the initial motion vector MV0. For example, MV0 can be determined in the encoder and decoder in the same way through template matching. Alternatively, the motion vector can be predicted as a function of the motion vectors of the neighboring blocks of the current block in the spatial or temporal domain.

[0221] The initial motion vector MV0 is the initial estimate of the final motion vector MV0”, which is used for inter-frame prediction of the current block. It constitutes the input to the correction process and outputs the final motion vector MV0” at the end of the correction process.

[0222] S602: Determine the search space location based on the initial motion vector.

[0223] Assume the search space location includes a central search space location and adjacent search space locations, where the initial motion vector points to the central search space location. In one feasible implementation, step S602 includes: determining the central search space location based on the initial motion vector; and determining adjacent search space locations based on one or more preset offsets and the central search space location.

[0224] In one feasible implementation, the search space consists of search space locations, and the pattern of the search space is a 5×5 square of search space locations.

[0225] For example, a search space is constructed based on an initial motion vector MV0, and one or more candidate motion vectors are associated with the initial motion vector. Then, a motion vector MV0 (corresponding to the coordinates of the search space position) is selected from the initial motion vector MV0 and one or more candidate motion vectors based on the matching cost. (S603) It should be noted that for some candidate motion vectors in the search space (which may be all candidate motion vectors in each partial search space determined for each stage in different embodiments of this application), the cost can be included as part of the search space construction and calculated during the search space construction.

[0226] In one feasible implementation, the candidate motion vector of the current block points from the top-left pixel of the current block in the current image to the corresponding top-left pixel of the candidate prediction block in the reference image. Therefore, the top-left pixel of the candidate prediction block represents the search space in the reference image. The top-left pixel of the block serves as the block's location. It should be noted that any other pixel of the block can serve as the block's location, and it should be understood that the same location convention applies to all blocks. For example, the motion vector can be equivalently defined as a movement from the center pixel of the current block to the center pixel of the corresponding candidate block.

[0227] It should be noted that each candidate motion vector, including the initial motion vector, points to a pixel in the reference image, i.e., a location in the search space. The coordinate relationship between the pixel pointed to by the initial motion vector and the pixels pointed to by other candidate motion vectors can be represented by one or more motion vector offsets between the initial motion vector and other candidate motion vectors.

[0228] One or more motion vector offsets can be predetermined. This means that the positional relationship between the central search space location and adjacent search space locations can be predetermined.

[0229] S603: Check the matching cost of the search space locations in the order of checks to select the target search space location with the minimum matching cost.

[0230] In one feasible implementation, the matching cost of each search space location in the search space is checked sequentially according to the checking order; the search space location with the minimum matching cost is selected as the target search space location.

[0231] Matching cost can be measured using a cost function. For example, the cost function could be the sum of the absolute differences between the template and a reference image region, where the reference image region corresponds to the template at the location pointed to by the candidate motion vector. After calculating the sum of the absolute differences (SAD) of all candidate motion vectors, the candidate motion vector with the smallest SAD is selected. It should be noted that SAD is just an example. The cost function can be SAD (sum of absolute differences), MRSAD (mean removed sum of absolute differences), SSE (sum of squared errors), or any other cost function used to represent similarity. The optimal motion vector is selected by comparing the matching costs at different search spatial locations.

[0232] In one feasible implementation, the matching cost of a search space location is compared with the template minimal matching cost; when the matching cost of a search space location is less than the template minimal matching cost, the matching cost of a search space location is set to the template minimal matching cost.

[0233] After examining the last search space location in the search space, set the minimum matching cost of the template to the minimum matching cost.

[0234] In one embodiment, such as Figure 7 As shown, the search space can be in a square pattern. For example, the pixel to which the initial motion vector points is considered to be (0, 0) in the coordinate system. In the coordinate system, horizontal to the right is considered the positive horizontal direction, and vertical downward is considered the positive vertical direction.

[0235] The search space can include 25 search space locations. The pixels with coordinates (–2, –2), (2, 2), (–2, 2), and (2, –2) are the corners of a square.

[0236] The 25 search space locations are divided into one or more groups to define the inspection order. It should be noted that, depending on the specific implementation, the step of dividing the search space locations into one or more groups may not be necessary; this is determined only to clearly describe the design of one inspection order.

[0237] In the first specific embodiment, the 25 search space locations are divided into two groups:

[0238] Group 1: (0, 0);

[0239] Group 2: All other search space locations.

[0240] like Figure 8 As shown, the number of each search space position indicates the order in which they are checked, with smaller numbers checked before larger numbers. The position marked "1" (0,0) is checked first, and then the other search space positions in the second group are checked in a horizontal order (from left to right, row by row). Therefore, the checking order is (0,0), (–2,–2), (–1,–2), (0,–2), (1,–2), (2,–2), (–2,–1), (–1,–1), (0,–1), (1,–1), (2,–1), (–2,0), (–1,0), (1,0), (2,0), (–2,1), (–1,1), (0,1), (1,1), (2,1), (–2,2), (–1,2), (0,2), (1,2), (2,2). It should be noted that the search space position in the second group can also be checked according to other inspection orders, such as vertical inspection order (from top to bottom, column by column), zigzag inspection order, etc.

[0241] In the current embodiment, the center position of the search space is checked first, and then other search space positions are checked according to a preset checking order. It can also be implemented based on a non-square pattern search space. For example, it could be a cross-shaped pattern with 21 search space positions, corresponding to... Figure 9 The inspection sequence is shown. For example, it could be a Union flag pattern with 17 search space positions, corresponding to... Figure 10 The inspection sequence is shown.

[0242] In a second specific embodiment, the second group can be further divided into more groups, for example, dividing the 25 search space locations into four groups:

[0243] Group 1: (0, 0);

[0244] The second group: (–1, 0), (0, 1), (1, 0), (0, –1) (adjacent positions of the center position in the vertical or horizontal direction);

[0245] The third group: (–1, –1), (–1, 1), (1, 1), (1, –1) (the sample whose position is 1 pixel away from the center position and is not included in the second group);

[0246] Group 4: All other search space locations.

[0247] The first, second, third, and fourth groups are checked sequentially. It should be noted that the checking order within the same group is not limited. For example, for the second group, the checking order could be (-1, 0), (0, 1), (1, 0), (0, -1) or (-1, 0), (1, 0), (0, 1), (0, -1). It should also be noted that the search space positions in the fourth group can be checked according to different checking orders (such as clockwise or counterclockwise) and different starting checking points (such as the top-left pixel of the search space or other positions in the fourth group). Figure 7 An example of this is the check order (0,0), (-1,0), (0,1), (1,0), (0,-1), (-1,-1), (-1,1), (1,1), (1,-1), (-2,0), (-2,1), (-2,2), (-1,2), (0,2), (1,2), (2,2), (2,1), (2,0), (2,-1), (2,-2), (1,-2), (0,-2), (-1,-2), (-2,-2), (-2,-1).

[0248] It can also be implemented based on other pattern search spaces. For example, it could be a rhombus pattern with 13 search space positions, corresponding to... Figure 11 The inspection sequence is shown.

[0249] In a third specific embodiment, the fourth group can be further divided into more groups, for example, dividing the 25 search space locations into five groups:

[0250] Group 1: (0, 0);

[0251] The second group: (–1, 0), (0, 1), (1, 0), (0, –1) (adjacent positions of the center position in the vertical or horizontal direction);

[0252] The third group: (–1, –1), (–1, 1), (1, 1), (1, –1) (the sample whose position is 1 pixel away from the center position and is not included in the second group);

[0253] Group 4: (–2, 0), (0, 2), (2, 0), (0, –2) (positions are 2 pixels away from the center in the vertical / horizontal direction and 0 pixels away from the center in the other horizontal / vertical direction);

[0254] Group 5: All other search space locations.

[0255] like Figure 12 As shown, for example, the check order is (0,0), (-1,0), (0,1), (1,0), (0,-1), (-1,-1), (-1,1), (1,1), (1,-1), (-2,0), (0,2), (2,0), (0,-2), (-2,-1), (-2,1), (-2,2), (-1,2), (1,2), (2,2), (2,1), (2,-1), (2,-2), (1,-2), (-1,-2), (-2,-2).

[0256] In the fourth specific embodiment, the fifth group can be further divided into more groups, for example, dividing the 25 search space locations into six groups:

[0257] Group 1: (0, 0);

[0258] The second group: (–1, 0), (0, 1), (1, 0), (0, –1) (adjacent positions of the center position in the vertical or horizontal direction);

[0259] The third group: (–1, –1), (–1, 1), (1, 1), (1, –1) (the sample whose position is 1 pixel away from the center position and is not included in the second group);

[0260] Group 4: (–2, 0), (0, 2), (2, 0), (0, –2) (positions are 2 pixels away from the center in the vertical / horizontal direction and 0 pixels away from the center in the other horizontal / vertical direction);

[0261] Group 5: (–2, –2), (–2, 2), (2, 2), (2, –2) (positions are 2 pixels away from the center in both the vertical and horizontal directions);

[0262] Group 6: All other search space locations.

[0263] like Figure 13 As shown, for example, the check order is (0,0), (-1,0), (0,1), (1,0), (0,-1), (-1,-1), (-1,1), (1,1), (1,-1), (-2,0), (0,2), (2,0), (0,-2), (-2,-2), (-2,2), (2,-2), (-2,-1), (-2,1), (-1,2), (1,2), (2,1), (2,-1), (1,-2), (-1,-2).

[0264] In the fifth specific embodiment, the 25 search space locations are not divided. The checks are performed according to a preset inspection order (e.g., a horizontal inspection order). Figure 14 As shown, for example, the check order is (–2, –2), (–1, –2), (0, –2), (1, –2), (2, –2), (–2, –1), (–1, –1), (0, –1), (1, –1), (2, –1), (–2, 0), (–1, 0), (0, 0), (1, 0), (2, 0), (–2, 1), (–1, 1), (0, 1), (1, 1), (2, 1), (–2, 2), (–1, 2), (0, 2), (1, 2), (2, 2).

[0265] S604: Determine the corrected motion vector of the current block based on the initial motion vector and the target search space position.

[0266] "Modified motion vector MV0" is the motion vector pointing to the target search space position. First, the offset between the target search space position and the center search space position can be derived. Then, by adding the initial motion vector and the offset, the modified motion vector MV0 can also be derived.

[0267] Figure 15 The inter-frame prediction device 1500 of this application is shown.

[0268] The inter-frame prediction device 1500 includes: an acquisition module 1501 for acquiring the initial motion vector of the current block; a setting module 1502 for determining the search space position based on the initial motion vector; a calculation module 1503 for checking the matching cost of the search space positions in the inspection order to select the target search space position with the minimum matching cost; and a prediction module 1504 for determining the corrected motion vector of the current block based on the initial motion vector and the target search space position, wherein the center search space position is checked first in the inspection order, and the initial motion vector points to the center search space position.

[0269] In one feasible implementation, the search space location includes a central search space location and adjacent search space locations, wherein the setting module 1502 is used to: determine the central search space location based on the initial motion vector; and determine adjacent search space locations based on one or more preset offsets and the central search space location.

[0270] In one feasible implementation, the search space consists of search space locations, and the pattern of the search space is a 5×5 search space location square.

[0271] In one feasible implementation, the calculation module 1503 is used to: sequentially check the matching cost of each search space location in the search space location according to the checking order; and select the search space location with the minimum matching cost from the search space locations as the target search space location.

[0272] In one feasible implementation, the calculation module 1503 is used to: compare the matching cost of a search space location in the search space with the minimum matching cost of the template; when the matching cost of a search space location in the search space is less than the minimum matching cost of the template, set the matching cost of a search space location in the search space as the minimum matching cost of the template; and after checking the last search space location in the search space, set the minimum matching cost of the template as the minimum matching cost.

[0273] In one feasible implementation, the central search space position is set to (0, 0) in the coordinate system, with the horizontal direction to the right set as the positive horizontal direction and the vertical direction downward set as the positive vertical direction.

[0274] In one feasible implementation, the check order is (0,0), (-2,-2), (-1,-2), (0,-2), (1,-2), (2,-2), (-2,-1), (-1,-1), (0,-1), (1,-1), (2,-1), (-2,0), (-1,0), (1,0), (2,0), (-2,1), (-1,1), (0,1), (1,1), (2,1), (-2,2), (-1,2), (0,2), (1,2), (2,2).

[0275] In one feasible implementation, the check order is (0,0), (-1,0), (0,1), (1,0), (0,-1), (-1,-1), (-1,1), (1,1), (1,-1), (-2,0), (-2,1), (-2,2), (-1,2), (0,2), (1,2), (2,2), (2,1), (2,0), (2,-1), (2,-2), (1,-2), (0,-2), (-1,-2), (-2,-2), (-2,-1).

[0276] In one feasible implementation, the check order is (0,0), (-1,0), (0,1), (1,0), (0,-1), (-1,-1), (-1,1), (1,1), (1,-1), (-2,0), (0,2), (2,0), (0,-2), (-2,-1), (-2,1), (-2,2), (-1,2), (1,2), (2,2), (2,1), (2,-1), (2,-2), (1,-2), (-1,-2), (-2,-2).

[0277] In one feasible implementation, the check order is (0,0), (-1,0), (0,1), (1,0), (0,-1), (-1,-1), (-1,1), (1,1), (1,-1), (-2,0), (0,2), (2,0), (0,-2), (-2,-2), (-2,2), (2,2), (2,-2), (-2,-1), (-2,1), (-1,2), (1,2), (2,1), (2,-1), (1,-2), (-1,-2).

[0278] Figure 16 The inter-frame prediction apparatus 1600 of this application may be a decoder or an encoder. The apparatus 1600 includes: one or more processors 1601; and a non-transitory computer-readable storage medium 1602 coupled to the processors and storing a program executed by the processors, wherein, when the processor executes the program, it causes the decoder to execute... Figure 6 The method.

[0279] In another embodiment, a computer program product including program code is provided, which, when executed by a computer or processor, is used to perform... Figure 6 The method shown.

[0280] In another embodiment, a non-transitory computer-readable medium carrying program code is provided, which, when executed by a computer device, causes the computer device to perform... Figure 6 The method.

[0281] The following section describes the application of the encoding and decoding methods shown in the above embodiments, as well as the systems using these methods.

[0282] Figure 17This is a block diagram of a content provisioning system 3100 for implementing a content distribution service. The content provisioning system 3100 includes a capture device 3102, a terminal device 3106, and optionally a display 3126. The capture device 3102 and the terminal device 3106 communicate via a communication link 3104. This communication link may include the aforementioned communication channel 13. The communication link 3104 includes, but is not limited to, Wi-Fi, Ethernet, cable, wireless (3G / 4G / 5G), USB, or any combination thereof.

[0283] The capture device 3102 generates data and can encode the data using the encoding method shown in the above embodiments. Alternatively, the capture device 3102 can distribute the data to a streaming server (not shown in the figure), which encodes the data and sends the encoded data to the terminal device 3106. The capture device 3102 includes, but is not limited to, cameras, smartphones or tablets, computers or laptops, video conferencing systems, PDAs, in-vehicle devices, or any combination thereof. For example, the capture device 3102 may include the source device 12 as described above. When the data includes video, the video encoder 20 included in the capture device 3102 can actually perform video encoding processing. When the data includes audio (i.e., voice), the audio encoder included in the capture device 3102 can actually perform audio encoding processing. In some practical scenarios, the capture device 3102 distributes the encoded video and audio data together. In other practical scenarios, such as in a video conferencing system, the encoded audio data and encoded video data are not multiplexed. The capture device 3102 distributes the encoded audio data and encoded video data separately to the terminal device 3106.

[0284] In the content delivery system 3100, the terminal device 310 receives and reproduces encoded data. The terminal device 3106 can be a device with data reception and reproduction capabilities, such as a smartphone or tablet 3108, a computer or laptop 3110, a network video recorder (NVR) / digital video recorder (DVR) 3112, a TV 3114, a set-top box (STB) 3116, a video conferencing system 3118, a video surveillance system 3120, a personal digital assistant (PDA) 3122, an in-vehicle device 3124, or any combination thereof, or a device capable of decoding the aforementioned encoded data. For example, the terminal device 3106 may include the destination device 14 as described above. When the encoded data includes video, the video decoder 30 included in the terminal device prioritizes video decoding. When the encoded data includes audio, the audio decoder included in the terminal device prioritizes audio decoding.

[0285] For terminal devices with displays, such as smartphones or tablets 3108, computers or laptops 3110, network video recorders (NVRs) / digital video recorders (DVRs) 3112, TVs 3114, personal digital assistants (PDAs) 3122, or in-vehicle devices 3124, the terminal device can feed the decoded data to its display. For terminal devices without displays (e.g., STBs 3116, video conferencing systems 3118, or video surveillance systems 3120), an external display 3126 is connected to receive and display the decoded data.

[0286] Each device in this system can use an image encoding device or an image decoding device as shown in the above embodiments when performing encoding or decoding.

[0287] Figure 18 This is a diagram illustrating an example structure of terminal device 3106. After terminal device 3106 receives a stream from capture device 3102, protocol proceeding unit 3202 analyzes the transport protocol of the stream. Protocols include, but are not limited to, Real-Time Streaming Protocol (RTSP), Hypertext Transfer Protocol (HTTP), HTTP Live Streaming Protocol (HLS), MPEG-DASH, Real-Time Transport Protocol (RTP), Real-Time Messaging Protocol (RTMP), or any combination thereof.

[0288] After processing the stream, the protocol processing unit 3202 generates a stream file. The file is then output to the demultiplexing unit 3204. The demultiplexing unit 3204 can separate the multiplexed data into encoded audio data and encoded video data. As mentioned above, in some practical scenarios, such as in a video conferencing system, encoded audio data and encoded video data are not multiplexed. In this case, the encoded data is sent to the video decoder 3206 and the audio decoder 3208 without going through the demultiplexing unit 3204.

[0289] Through demultiplexing, a video elementary stream (ES), an audio ES, and optional subtitles are generated. Video decoder 3206 includes video decoder 30 as described in the above embodiments, which decodes the video ES to generate video frames using the decoding method shown in the above embodiments, and feeds the data to synchronization unit 3212. Audio decoder 3208 decodes the audio ES to generate audio frames, and feeds the data to synchronization unit 3212. Alternatively, the video frame can be stored in a buffer before being fed to synchronization unit 3212. Figure 18 (Not shown in the image). Similarly, before feeding the audio frame to the synchronization unit 3212, the audio frame can be stored in a buffer (not shown in the image). Figure 18 (Not shown in the text)

[0290] Synchronization unit 3212 synchronizes video and audio frames and provides the video / audio to video / audio display 3214. For example, synchronization unit 3212 synchronizes the presentation of video and audio information. Information can be decoded in the syntax using timestamps about the representation of the decoded audio and video data and timestamps about the transmission of the data stream itself.

[0291] If the stream includes subtitles, the subtitle decoder 3210 decodes the subtitles, synchronizes the subtitles with the video and audio frames, and provides the video / audio / subtitles to the video / audio / subtitle display 3216.

[0292] This application is not limited to the above-described system. The image encoding device or image decoding device in the above embodiments can be integrated into other systems, such as automotive systems.

[0293] Mathematical operators

[0294] The mathematical operators used in this application are similar to those used in the C programming language. However, the results of integer division and arithmetic shift operations are precisely defined here, and other operations such as exponentiation and real-value division are also defined. Numbering and counting conventions are generally agreed to start from 0; for example, "first" is equivalent to the 0th, "second" is equivalent to the 1st, and so on.

[0295] Arithmetic operators

[0296] The following arithmetic operators are defined as follows:

[0297] Addition

[0298] – Subtraction (used as a two-parameter operator) or NOT (used as a unary prefix operator).

[0299] * Multiplication, including matrix multiplication.

[0300]

[0301] ÷ is used to represent division in mathematical equations, but it does not have truncation or rounding operations.

[0302] It is used to represent division operations in mathematical equations, but without truncation or rounding.

[0303] The sum of f(i), where i takes all integer values ​​from x to y, inclusive.

[0304] The x%y modulo operation is the remainder when x is divided by y. Here, both x and y must be integers, x≥0 and y>0.

[0305] Logical operators

[0306] The following logical operators are defined as follows:

[0307] x&&y Boolean logical AND operation between x and y

[0308] Boolean logical OR operation between x and y || y

[0309] ! Boolean logic "NOT"

[0310] x? y:z If x is true or not equal to 0, then x equals the value of y; otherwise, x equals the value of z.

[0311] Relational operators

[0312] The following relational operators are defined as follows:

[0313] > Greater than

[0314] >= Greater than or equal to

[0315] < Less than

[0316] <= Less than or equal to

[0317] == equals

[0318] ! = Not equal to

[0319] When a relational operator is used on a syntax element or variable that has been assigned the value "na" (not applicable), the value "na" is treated as the unique value of that syntax element or variable. The value "na" is not equal to any other value.

[0320] Bitwise operators

[0321] The bitwise operators are defined as follows:

[0322] & is a bitwise AND operation. When performing operations on integer parameters, it operates on the two's complement representation of the integer value.

[0323] When operating on binary arguments, if it contains fewer bits than the other argument, the shorter argument is extended by adding more significant bits equal to 0.

[0324] | Bitwise "OR". When operating on integer arguments, the two's complement representation of the integer value is used.

[0325] When operating on binary arguments, if it contains fewer bits than the other argument, the shorter argument is extended by adding more significant bits equal to 0.

[0326] ^ Bitwise "XOR". When operating on integer arguments, the two's complement representation of the integer value is used.

[0327] When operating on binary arguments, if it contains fewer bits than the other argument, the shorter argument is extended by adding more significant bits equal to 0.

[0328] x >> y shifts the two's complement integer representation of x by y binary digits to the right. This function is only defined for non-negative integer values of y. The result of the right shift is that the value of the bit shifted into the most significant bit (MSB) is equal to the MSB of x before the shift operation.

[0329] x << y shifts the two's complement integer representation of x by y binary digits to the left. This function is only defined for non-negative integer values of y. The result of the left shift is that the value of the bit shifted into the least significant bit (LSB) is equal to 0.

[0330] Assignment operators

[0331] The following arithmetic operators are defined as follows:

[0332] = Assignment operator

[0333] ++ Increment, i.e., x++ is equivalent to x = x + 1; when used in array indexing, it is equal to the value of the variable before the increment operation.

[0334] –– Decrement, i.e., x–– is equivalent to x = x – 1; when used in array indexing, it is equal to the value of the variable before the decrement operation.

[0335] += Increment by a specified amount, i.e., x += 3 is equivalent to x = x + 3, x += (–3) is equivalent to x = x +

[0336] (–3).

[0337] –= Decrement by a specified amount, i.e., x –= 3 is equivalent to x = x – 3, x –= (–3) is equivalent to x = x – (–3).

[0338] Range representation

[0339] The following notation is used to specify a range of values:

[0340] x = y..z x takes integer values ​​from y to z (inclusive), where x, y, and z are integers, and z is greater than y.

[0341] Mathematical functions

[0342] The mathematical function is defined as follows:

[0343]

[0344] Asin(x) is the trigonometric arcsine function, operated on with respect to the parameter x, where x is in the range of -1.0 to 1.0 (inclusive).

[0345] The output value is in the range of –π÷2 to π÷2 (inclusive), and the unit is radians.

[0346] Atan(x) is the trigonometric arctangent function. It operates on the parameter x and outputs a value in the range of –π÷2 to π÷2 (inclusive), in radians.

[0347]

[0348] Ceil(x) represents the smallest integer value greater than or equal to x.

[0349] Clip1 Y (x) = Clip3(0, (1< <BitDepth Y )–1,x)

[0350] Clip1 C (x) = Clip3(0, (1< <BitDepth C )–1,x)

[0351]

[0352] Cos(x) is a trigonometric cosine function that operates on the parameter x, with units of radians.

[0353] Floor(x) represents the largest integer value less than or equal to x.

[0354]

[0355] The natural logarithm of Ln(x)x ​​(the logarithm to the base e, where e is the constant of the base of the natural logarithm, 2.718281828...).

[0356] Log2(x) is the logarithm of x to the base 2.

[0357] Log10(x) is the logarithm of x to the base 10.

[0358]

[0359] Round(x)=Sign(x)*Floor(Abs(x)+0.5)

[0360]

[0361] Sin(x) is a trigonometric sine function that operates on the parameter x, with units of radians.

[0362]

[0363] Swap(x, y) = (y, x)

[0364] Tan(x) is the trigonometric tangent function, which operates on the parameter x and is expressed in radians.

[0365] Operation priority order

[0366] When parentheses are not used to explicitly indicate precedence, the following rule applies:

[0367] - Higher priority operations are performed before lower priority operations.

[0368] Operations of the same priority are performed from left to right.

[0369] The table below shows the order of operations from highest to lowest. The higher the position in the table, the higher the priority.

[0370] If these operators are also used in the C programming language, the precedence order used in this article is the same as that used in the C programming language.

[0371] Table: Operation priority is sorted from highest (top of table) to lowest (bottom of table).

[0372]

[0373] Textual description of logical operations

[0374] In the text, the statements in logical operations are described in mathematical form as follows:

[0375]

[0376]

[0377] It can be described in the following way:

[0378] ...as follows / ...the following shall prevail:

[0379] - If condition 0, then statement 0.

[0380] Otherwise, if condition 1 is true, then statement 1.

[0381] -……

[0382] – Otherwise (a hint about the remaining conditions), then statement n

[0383] Each "if...otherwise, if...otherwise,..." statement in the text is introduced by "...as follows" or "...as follows" (followed by "if..."). The last condition "if...otherwise, if...otherwise,..." always has an "otherwise,...". Inserted "if...otherwise, if...otherwise,..." statements can be identified by matching "...as follows" or "...as follows" and ending with "otherwise,...".

[0384] In the text, the statements in logical operations are described in mathematical form as follows:

[0385]

[0386] It can be described in the following way:

[0387] ...as follows / ...the following shall prevail:

[0388] – Statement 0 is true if all of the following conditions are true:

[0389] –Condition 0a

[0390] –Condition 0b

[0391] Otherwise, if one or more of the following conditions are true, then statement 1:

[0392] –Condition 1a

[0393] –Condition 1b

[0394] -……

[0395] Otherwise, statement n

[0396] In the text, the statements in logical operations are described in mathematical form as follows:

[0397]

[0398] It can be described in the following way:

[0399] If condition 0, then statement 0.

[0400] If condition 1 is true, then statement 1

[0401] Embodiments of encoder 20 and decoder 30, and the functions described herein relating to encoder 20 and decoder 30, can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on a computer-readable medium or transmitted as one or more instructions or code via a communication medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium, a tangible medium such as a data storage medium, or a communication medium that facilitates the transfer of a computer program from one place to another, for example, according to a communication protocol. In this manner, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium, such as a signal or carrier wave. A data storage medium may be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described herein. Computer program products may include computer-readable media.

[0402] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store required program code in the form of instructions or data structures and is accessible by a computer. Furthermore, any connection can be referred to as computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source via coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (e.g., infrared, radio, microwave, etc.), the definition of medium includes coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology (e.g., infrared, radio, and microwave, etc.). However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather refer to non-transient tangible storage media. Disks and optical discs as used herein include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where discs typically reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0403] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the above-described structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein can be provided in dedicated hardware and / or software modules for encoding and decoding, or incorporated into combined codecs. Furthermore, these techniques can be fully implemented in one or more circuit or logic elements.

[0404] The technology of this invention can be implemented in a variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or a set of ICs (e.g., chipsets). This invention describes various components, modules, or units to emphasize functional aspects of the apparatus used to perform the disclosed technology, but these components, modules, or units are not necessarily required to be implemented by different hardware units. Rather, as described above, various units can be combined in a codec hardware unit, or provided by a collection of interoperable hardware units including one or more processors as described above, combined with suitable software and / or firmware.

Claims

1. An inter-frame prediction method, characterized in that, include: Obtain the initial motion vector of the current block, which corresponds to the motion vector derived from the index; The search space location is determined based on the initial motion vector; The matching costs of the search space locations are checked in the order of inspection to select the target search space location with the minimum matching cost. Based on the initial motion vector and the target search space position, the corrected motion vector of the current block is determined, wherein the center search space position is checked first according to the checking order, and the initial motion vector points to the center search space position; The central search space position is set to (0, 0) in the coordinate system, with the horizontal direction to the right being the positive horizontal direction and the vertical direction downwards being the positive vertical direction. Check search space position (0, 0) before search space position (-2, -2), check search space position (-2, -2) before search space position (-1, -2), retrieve space position (-1, -2) before search space position (0, -2), check search space position (0, -2) before search space position (1, -2), check search space position (1, -2) before search space position (2, -2), check search space position (2, -1) before search space position (-2, -1). Before the search space position (-1, -1), check the search space position (-2, -1), before the search space position (0, -1), before the search space position (0, -1), before the search space position (1, -1), before the search space position (1, -1), before the search space position (2, -1), before the search space position (-2, -1), before the search space position (-1, -1), before the search space position (-2, -1), before the search space position (-1, 0), check the search space position (-2, -1), before the search space position (-1, 0). The search space position is (-2, 0). Before the search space position (1, 0), check the search space position (-1, 0). Before the search space position (2, 0), check the search space position (1, 0). Before the search space position (-2, 1), check the search space position (2, 0). Before the search space position (-1, 1), check the search space position (-2, 1). Before the search space position (-1, 1), check the search space position (-1, 1). Before the search space position (0, 1), check the search space position (-1, 1). (0,1), check search space position (1,1) before search space position (2,1), check search space position (2,1) before search space position (-2,2), check search space position (-2,2) before search space position (-1,2), check search space position (-1,2) before search space position (0,2), check search space position (0,2) before search space position (1,2), check search space position (1,2) before search space position (2,2).

2. The method according to claim 1, characterized in that, The search space location includes the central search space location and adjacent search space locations, wherein determining the search space location based on the initial motion vector includes: The location of the center search space is determined based on the initial motion vector; The adjacent search space positions are determined based on one or more preset offsets and the center search space position.

3. The method according to claim 2, characterized in that, The search space is composed of the search space locations, and the pattern of the search space is a 5×5 square of search space locations.

4. The method according to claim 1, characterized in that, The matching costs of the search space locations are checked according to the aforementioned checking order to select the target search space location with the minimum matching cost, including: According to the inspection order, the matching cost of each search space location in the search space location is checked in turn; The search space location with the minimum matching cost is selected from the search space locations and used as the target search space location.

5. The method according to claim 4, characterized in that, The matching cost of checking each search space location in the search space location according to the aforementioned checking order includes: The matching cost of one of the search space locations is compared with the template minimum matching cost. When the matching cost of one of the search space locations is less than the minimum matching cost of the template, the matching cost of one of the search space locations is set as the minimum matching cost of the template. After the last search space location in the search space is checked, the minimum matching cost of the template is set to the minimum matching cost.

6. An inter-frame prediction device, characterized in that, include: The acquisition module is used to acquire the initial motion vector of the current block, which corresponds to the motion vector derived from the index. The setting module is used to determine the search space location based on the initial motion vector; The calculation module is used to check the matching cost of the search space locations in the order of inspection in order to select the target search space location with the minimum matching cost. The prediction module is used to determine the corrected motion vector of the current block based on the initial motion vector and the target search space position, wherein the center search space position is checked first according to the checking order, and the initial motion vector points to the center search space position; The central search space position is set to (0, 0) in the coordinate system, with the horizontal direction to the right being the positive horizontal direction and the vertical direction downwards being the positive vertical direction. Check search space position (0, 0) before search space position (-2, -2), check search space position (-2, -2) before search space position (-1, -2), retrieve space position (-1, -2) before search space position (0, -2), check search space position (0, -2) before search space position (1, -2), check search space position (1, -2) before search space position (2, -2), check search space position (2, -1) before search space position (-2, -1). Before the search space position (-1, -1), check the search space position (-2, -1), before the search space position (0, -1), before the search space position (0, -1), before the search space position (1, -1), before the search space position (1, -1), before the search space position (2, -1), before the search space position (-2, -1), before the search space position (-1, -1), before the search space position (-2, -1), before the search space position (-1, 0), check the search space position (-2, -1), before the search space position (-1, 0). The search space position is (-2, 0). Before the search space position (1, 0), check the search space position (-1, 0). Before the search space position (2, 0), check the search space position (1, 0). Before the search space position (-2, 1), check the search space position (2, 0). Before the search space position (-1, 1), check the search space position (-2, 1). Before the search space position (-1, 1), check the search space position (-1, 1). Before the search space position (0, 1), check the search space position (-1, 1). (0,1), check search space position (1,1) before search space position (2,1), check search space position (2,1) before search space position (-2,2), check search space position (-2,2) before search space position (-1,2), check search space position (-1,2) before search space position (0,2), check search space position (0,2) before search space position (1,2), check search space position (1,2) before search space position (2,2).

7. The apparatus according to claim 6, characterized in that, The search space location includes the central search space location and adjacent search space locations, wherein the setting module is used for: The location of the center search space is determined based on the initial motion vector; The adjacent search space positions are determined based on one or more preset offsets and the center search space position.

8. The apparatus according to claim 7, characterized in that, The search space is composed of the search space locations, and the pattern of the search space is a 5×5 square of search space locations.

9. The apparatus according to claim 6, characterized in that, The calculation module is used for: According to the inspection order, the matching cost of each search space location in the search space location is checked in turn; The search space location with the minimum matching cost is selected from the search space locations and used as the target search space location.

10. The apparatus according to claim 9, characterized in that, The calculation module is used for: The matching cost of one of the search space locations is compared with the minimum matching cost of the template. When the matching cost of one of the search space locations is less than the minimum matching cost of the template, the matching cost of one of the search space locations is set as the minimum matching cost of the template. After checking the last search space position in the search space, the minimum matching cost of the template is set to the minimum matching cost.

11. A computer program product, characterized in that, Includes program code, which, when run on a processor, is used to perform the method according to any one of claims 1 to 5.

12. A decoder, characterized in that, include: One or more processors; A non-transitory computer-readable storage medium coupled to the processor and storing a program executed by the processor, wherein when the processor executes the program, the decoder performs the method according to any one of claims 1 to 5.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program and a bit stream, which, when executed by one or more processors, implements the method of any one of claims 1 to 5 to generate the bit stream.

Images