Video encoding and decoding method, device, medium and product

Abstract

The invention provides a video encoding and decoding method, equipment, a medium and a product. The method comprises the following steps: obtaining a spatially adjacent reference sample point of a current block; determining a first intra-frame prediction signal by using an intra-frame prediction mode based on a neural network based on the spatial adjacent reference sample points of the current block; and using the first intra prediction signal as an intra prediction signal or a candidate intra prediction signal for one or more prediction tools.

Description

Technical Field

[0001] This application relates to video encoding / decoding and compression. More specifically, this application relates to a method and apparatus for intra-frame prediction based on neural networks. Background Technology

[0002] Various electronic devices (such as digital televisions, laptops or desktop computers, tablets, digital cameras, digital recording devices, digital media players, video game consoles, smartphones, video conferencing equipment, video streaming devices, etc.) support digital video. Electronic devices send and receive, or otherwise transmit, digital video data via communication networks, and / or store digital video data on storage devices. Due to the limited bandwidth capacity of communication networks and the limited memory resources of storage devices, video data can be compressed using one or more video codec standards before it is transmitted or stored. Examples of video codec standards include Universal Video Codec (VVC), Joint Explore Test Model (JEM), High Efficiency Video Codec (HEVC / H.265), Advanced Video Codec (AVC / H.264), Moving Picture Experts Group (MPEG) codec, etc. Video codecs typically employ prediction methods that utilize the inherent redundancy in video data (e.g., inter-frame prediction, intra-frame prediction, etc.). Video codecs aim to compress video data into a form using a lower bitrate while avoiding or minimizing degradation in video quality. Summary of the Invention

[0003] According to a first aspect of this disclosure, a method for video decoding is provided, comprising: acquiring spatially nearest reference samples of a current block; determining a first intra-prediction signal using a neural network-based intra-prediction mode based on the spatially nearest reference samples of the current block; and using the first intra-prediction signal as an intra-prediction signal or a candidate intra-prediction signal for one or more prediction tools.

[0004] According to a second aspect of this disclosure, a method for video decoding is provided, comprising: determining a weight matrix for spatially nearest reference samples of a current block; determining a first intra-prediction signal based on the weight matrix and the spatially nearest reference samples; and replacing the intra-prediction signal determined using the one or more intra-prediction modes with the first intra-prediction signal in response to one or more intra-prediction modes being equal to the intra-prediction modes for the weight matrix during a decoder-side intra-mode derivation DIMD fusion process.

[0005] According to a third aspect of this disclosure, a method for video coding is provided, comprising: acquiring spatially nearest reference samples of a current block; determining a first intra-prediction signal using a neural network-based intra-prediction mode based on the spatially nearest reference samples of the current block; and using the first intra-prediction signal as an intra-prediction signal or a candidate intra-prediction signal for one or more prediction tools.

[0006] According to a fourth aspect of this disclosure, a method for video coding is provided, comprising: determining a weight matrix for spatially nearest reference samples of a current block; determining a first intra-prediction signal based on the weight matrix and the spatially nearest reference samples; and replacing the intra-prediction signal determined using the one or more intra-prediction modes with the first intra-prediction signal in response to one or more intra-prediction modes being equal to the intra-prediction modes for the weight matrix during a decoder-side intra-mode derivation DIMD fusion process.

[0007] It should be understood that the foregoing general description and the following detailed description are merely examples and not limitations of this disclosure. Attached Figure Description

[0008] Examples consistent with this disclosure are illustrated in conjunction with the accompanying drawings, which are included in and form part of this specification, and together with the description, serve to explain the principles of this disclosure.

[0009] Figure 1 This is a block diagram illustrating an exemplary system for encoding and decoding video blocks according to some embodiments of the present disclosure.

[0010] Figure 2 This is a block diagram illustrating an exemplary video encoder according to some embodiments of the present disclosure.

[0011] Figure 3 This is a block diagram illustrating an exemplary video decoder according to some embodiments of the present disclosure.

[0012] Figures 4A to 4E This is a block diagram illustrating how a frame is recursively divided into multiple video blocks of different sizes and shapes according to some embodiments of this disclosure.

[0013] Figure 5 The diagram illustrates a simplified representation of intra-frame modes as defined in VVC.

[0014] Figure 6 The diagram illustrates a simplified representation of multiple reference rows used for intra-frame prediction.

[0015] Figure 7A and Figure 7B The diagrams illustrate simplified reference samples for Position-Related Intra-Prediction Combination (PDPC) in the upper-right diagonal mode and the lower-left diagonal mode, respectively.

[0016] Figure 8A The diagram shows 4 × 8 and 8 A simplified diagram of the sub-partitions of CU 4.

[0017] Figure 8B The diagram illustrates the division by 4. 8, 8 4 and 4 A simplified diagram of the sub-partitions of the CU other than the CU in section 4.

[0018] Figure 9 The diagram illustrates the locations of the left and upper samples of the CU involved in the cross-component linear model (CCLM) prediction.

[0019] Figure 10 A simplified diagram of the matrix-weighted intra-frame prediction (MIP) process is shown.

[0020] Figure 11 The diagram illustrates a simplified representation of the L-shaped neighborhood of a given prediction block.

[0021] Figure 12 The diagram illustrates a simplified representation of the prediction of Y from the context X of reference samples surrounding the current w × h block Y via a neural network-based intra-frame prediction mode. Here, w = 8 and h = 4.

[0022] Figure 13 The diagram shows that Current block Context of surrounding reference samples Decomposed into usable reference samples and unavailable reference points A simplified diagram. Here, and In the case illustrated, the number of unavailable reference points cannot reach its maximum value.

[0023] Figure 14 The diagram illustrates a simplified representation of the intra-prediction mode signaling for the current w × h luminance CB, outlined in orange dashed lines. The pixel coordinates of the top-left corner of this CB are (y, x). The binary values ​​of the nnFlag are shown in bold gray. Here, h = 8, w = 4, x = 8, and y = 0.

[0024] Figure 15 The illustration shows the selection of unidirectional predictive motion vectors in the Geometric Partitioning Mode (GPM).

[0025] Figure 16 The diagram illustrates the top neighbor block and left neighbor block used in the CIIP weight derivation.

[0026] Figure 17The diagram illustrates the method for dividing angle patterns.

[0027] Figure 18 The diagram illustrates GPM using inter-frame and intra-frame prediction. Available IPM candidates are shown in (a) to (c). (d) is an example of GPM using intra-frame and intra-frame prediction.

[0028] Figure 19 The image shows the edges on the template.

[0029] Figure 20 The figure illustrates the ramp function for the weights of GPM mixing based on the displacement (d) from the predicted sample location to the GPM partition boundary and the mixing region size (τ).

[0030] Figure 21 The illustration shows the spatial GPM candidates.

[0031] Figure 22 The image shows the GPM template.

[0032] Figure 23 The diagram illustrates GPM mixing.

[0033] Figure 24 This is a simplified diagram illustrating a computing environment coupled to a user interface according to some embodiments of the present disclosure.

[0034] Figure 25 This is a flowchart illustrating a video decoding method according to some embodiments of the present disclosure.

[0035] Figure 26 This is a flowchart illustrating a video decoding method according to some embodiments of the present disclosure.

[0036] Figure 27 This is a flowchart illustrating a video encoding method according to some embodiments of the present disclosure.

[0037] Figure 28 This is a flowchart illustrating a video encoding method according to some embodiments of the present disclosure. Detailed Implementation

[0038] Referring now to the detailed description, examples of which are illustrated in the accompanying drawings. Numerous non-limiting details are set forth in the following detailed description to aid in understanding the subject matter presented herein. However, various alternatives may be used without departing from the scope of the claims, and the subject matter may be practiced without these specific details. For example, the subject matter presented herein can be implemented on many classes of electronic devices with digital video capabilities.

[0039] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this disclosure are used to distinguish objects and are not used to describe any specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in sequences other than those shown in the drawings or described in this disclosure.

[0040] Figure 1 This is a block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel, according to some embodiments of the present disclosure. Figure 1 As shown, system 10 includes a source device 12 that generates and encodes video data that will later be decoded by a target device 14. The source device 12 and target device 14 can include any electronic device from a wide variety of electronic devices, including cloud servers, server computers, desktop or laptop computers, tablet computers, smartphones, set-top boxes, digital televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some embodiments, the source device 12 and target device 14 are equipped with wireless communication capabilities.

[0041] In some implementations, target device 14 may receive encoded video data to be decoded via link 16. Link 16 may include any type of communication medium or device capable of moving encoded video data from source device 12 to target device 14. In one example, link 16 may include a communication medium enabling source device 12 to transmit encoded video data directly to target device 14 in real time. The encoded video data may be modulated according to communication standards, such as wireless communication protocols, and transmitted to target device 14. The communication medium may include any wireless or wired communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include a router, a switch, a base station, or any other means that may facilitate communication from source device 12 to target device 14.

[0042] In some other implementations, encoded video data can be sent from output interface 22 to storage device 32. The target device 14 can then access the encoded video data in storage device 32 via input interface 28. Storage device 32 can include any data storage medium of various distributed or locally accessed data storage media, such as hard disk drives, Blu-ray discs, digital universal discs (DVDs), compact disc read-only memory (CD-ROMs), flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In another example, storage device 32 can correspond to a file server or another intermediate storage device that can hold the encoded video data generated by source device 12. Target device 14 can access the stored video data from storage device 32 via streaming or downloading. The file server can be any type of computer capable of storing and sending encoded video data to target device 14. Exemplary file servers include web servers (e.g., for websites), file transfer protocol (FTP) servers, network attached storage (NAS) devices, or local disk drives. Target device 14 can access the encoded video data via any standard data connection suitable for accessing encoded video data stored on the file server. Standard data connections include wireless channels (e.g., Wi-Fi connections), wired connections (e.g., digital subscriber line (DSL), cable modems, etc.), or a combination of both. Transfer of encoded video data from storage device 32 can be streaming, downloading, or a combination of both.

[0043] like Figure 1 As shown, source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Video source 18 may include sources or combinations of such sources, such as: a video capture device (e.g., a camera), a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as source video. As an example, if video source 18 is a camera in a security monitoring system, source device 12 and target device 14 may form a camera phone or video phone. However, the embodiments described in this application are generally applicable to video encoding and decoding and can be applied to wireless and / or wired applications.

[0044] Video, whether captured, pre-captured, or computer-generated, can be encoded by video encoder 20. The encoded video data can be sent directly to target device 14 via output interface 22 of source device 12. Alternatively, the encoded video data can be stored on storage device 32 for later access by target device 14 or other devices for decoding and / or playback. Output interface 22 may further include a modem and / or transmitter. The encoded video data may include a sequence of images, each image comprising one or more sample arrays, for example, luminance only (Y) for a monochrome; luminance and two chrominances in the YCbCr or YCgCo domain; or green, blue, and red in the GBR (also known as RGB) domain. For ease of reference and terminology in this application, in some embodiments, the variables and terms associated with each set having three sample arrays may be referred to as luminance and chrominance, where the two chrominance arrays may be referred to as Cb and Cr, regardless of the actual color representation used. Video data can be in chroma format 4:0:0, chroma format 4:2:0, chroma format 4:2:2 or chroma format 4:4:4, but this application is not limited to these.

[0045] Target device 14 includes an input interface 28, a video decoder 30, and a display device 34. Input interface 28 may include a receiver and / or a modem, and receives encoded video data via link 16. The encoded video data transmitted via link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 when decoding the video data. Such syntax elements may be included within encoded video data transmitted on a communication medium, stored on a storage medium, or stored on a file server.

[0046] In some embodiments, the target device 14 may include a display device 34, which may be an integrated display device or an external display device configured to communicate with the target device 14. The display device 34 displays decoded video data to a user and may include any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0047] Video encoder 20 and video decoder 30 can operate according to proprietary or industry standards (e.g., VVC, HEVC, MPEG-4 Part 10, AVC) or extensions of such standards. It should be understood that this application is not limited to any particular video encoding / decoding standard and can be applied to other video encoding / decoding standards. It is generally understood that the video encoder 20 of source device 12 can be configured to encode video data according to any of these current or future standards. Similarly, it is also generally understood that the video decoder 30 of target device 14 can be configured to decode video data according to any of these current or future standards.

[0048] The video encoder 20 and video decoder 30 can be implemented as any circuit of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic devices, software, hardware, firmware, or any combination thereof. When partially implemented in software, the electronic device may store instructions for software in a suitable non-transitory computer-readable medium and use one or more processors to execute the instructions in the hardware to perform the video encoding / decoding operations disclosed in this disclosure. Each of the video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder / decoder (CODEC) in the respective device.

[0049] In some implementations, components of source device 12 (e.g., video source 18, video encoder 20, or the following references) Figure 2 The components described include at least a portion of the components in the video encoder 20 and the output interface 22) and / or the components of the target device 14 (e.g., the input interface 28, the video decoder 30, or the following references). Figure 3At least a portion of the components included in the video decoder 30 and the display device 34 described herein can operate in a cloud computing service network such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS), where the cloud computing service network can provide software, platform, and / or infrastructure. In some embodiments, one or more components of the source device 12 and / or the target device 14 not included in the cloud computing service network may be located in one or more client devices, and these client devices may communicate with server computers in the cloud computing service network via wireless communication networks (e.g., cellular communication networks, short-range wireless communication networks, or Global Navigation Satellite System (GNSS) communication networks) or wired communication networks (e.g., local area network (LAN) communication networks or power line communication (PLC) networks). In embodiments, at least a portion of the operations described herein may be implemented as cloud-based services provided by one or more server computers, wherein the one or more server computers are implemented by at least a portion of the components of the source device 12 and / or at least a portion of the components of the target device 14 in the cloud computing service network; and one or more other operations described herein may be implemented by one or more client devices. In some implementations, the cloud computing service network may be a private cloud, a public cloud, or a hybrid cloud. Without departing from the scope of this disclosure, terms such as “cloud,” “cloud computing,” and “cloud-based” are used interchangeably as appropriate. It should be understood that this disclosure is not limited to implementation within the aforementioned cloud computing service networks. Rather, this disclosure may be implemented in any other type of computing environment currently known or developed in the future.

[0050] Figure 2 This is a block diagram illustrating an exemplary video encoder 20 according to some embodiments described in this application. The video encoder 20 can perform intra-frame predictive coding and inter-frame predictive coding on video blocks within a video frame. Intra-frame predictive coding relies on spatial prediction to reduce or remove spatial redundancy in the video data within a given video frame or picture. Inter-frame predictive coding relies on temporal prediction to reduce or remove temporal redundancy in the video data within neighboring video frames or pictures of a video sequence. It should be noted that in the field of video encoding and decoding, the term "frame" can be used as a synonym for the terms "image" or "picture".

[0051] like Figure 2As shown, the video encoder 20 includes a video data memory 40, a prediction processing unit 41, a decoded picture buffer (DPB) 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 further includes a motion estimation unit 42, a motion compensation unit 44, a segmentation unit 45, an intra-frame prediction processing unit 46, and an intra-frame block copying (BC) unit 48. In some embodiments, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and an adder 62 for video block reconstruction. A loop filter 63, such as a deblocking filter, can be located between the adder 62 and the DPB 64 to filter block boundaries to remove block artifacts from the reconstructed video. In addition to the deblocking filter, another loop filter (e.g., a sample adaptive offset (SAO) filter, a cross-component sample adaptive offset (CCSAO) filter, and / or an adaptive loop filter (ALF)) can be used to filter the output of the adder 62. It should be noted that, regarding the CCSAO technique, this application is not limited to the embodiments described herein, but can also be applied to situations where an offset is selected for any other component of the luminance component and the two chrominance components based on any one of the luminance component and the two chrominance components (as mentioned above, for ease of notation and terminology in this application, in the YCbCr domain, the luminance component and the two chrominance components can represent Y, Cb, and Cr; in the YCgCo domain, the luminance component and the two chrominance components can represent Y, Cg, and Co; or in the RGB domain, the luminance component and the two chrominance components can represent G, B, and R) to modify that other component based on the selected offset. Furthermore, it should be noted that the first component mentioned herein can be any one of the luminance component and the two chrominance components, the second component mentioned herein can be any one of the luminance component and the two chrominance components, and the third component mentioned herein can be the remaining component of the luminance component and the two chrominance components. In some examples, the loop filter can be omitted, and the decoded video block can be directly provided to the DPB 64 by the adder 62. The video encoder 20 may take the form of a fixed or programmable hardware unit, or may be distributed among one or more of the fixed or programmable hardware units described.

[0052] The video data storage device 40 can store video data encoded by the components of the video encoder 20. For example, it can store data from... Figure 1The video source 18 shown receives video data from the video data memory 40. The DPB 64 is a buffer that stores reference video data (e.g., reference frames or pictures) used by the video encoder 20 (e.g., in intra-frame or inter-frame predictive coding modes) when encoding the video data. The video data memory 40 and DPB 64 can be formed from any of a variety of memory devices. In various examples, the video data memory 40 may be on-chip along with other components of the video encoder 20, or off-chip relative to those components.

[0053] like Figure 2 As shown, after receiving video data, the segmentation unit 45 within the prediction processing unit 41 segments the video data into video blocks. This segmentation may also include segmenting the video frame into strips, tiles (e.g., a collection of video blocks) or other larger coding units (CUs) according to a predefined splitting structure associated with the video data (e.g., a quadtree (QT) structure). A video frame is, or can be considered, a two-dimensional array or matrix of sample points with sample values. Sample points in the array may also be referred to as pixels or image elements (pel). The number of sample points in the horizontal and vertical directions (or axes) of the array or image defines the size and / or resolution of the video frame. For example, a video frame can be divided into multiple video blocks using QT segmentation. A video block is again, or can be considered, a two-dimensional array or matrix of sample points with sample values, but its dimension is smaller than that of the video frame. The number of sample points in the horizontal and vertical directions (or axes) of the video block defines the size of the video block. Video blocks can be further divided into one or more block partitions or sub-blocks (which can then re-form blocks) by iteratively using, for example, QT partitioning, binary tree (BT) partitioning, or ternary tree (TT) partitioning, or any combination thereof. It should be noted that the term "block" or "video block" as used herein can be a portion of a frame or image, particularly a rectangular (square or non-square) portion. Referring, for example, to HEVC and VVC, a block or video block can be or corresponds to a coding tree unit (CTU), CU, prediction unit (PU), or transform unit (TU) and / or can be or corresponds to a corresponding block (e.g., coding tree block (CTB), coding block (CB), prediction block (PB), or transform block (TB)) and / or sub-block.

[0054] The prediction processing unit 41 can select one of several feasible predictive coding modes for the current video block based on error results (e.g., coding rate and distortion level), such as one or more inter-frame predictive coding modes among multiple intra-frame predictive coding modes. The prediction processing unit 41 can provide the resulting intra-frame or inter-frame predictive coding block to adder 50 to generate a residual block, and to adder 62 to reconstruct the coding block for subsequent use as part of a reference frame. The prediction processing unit 41 also provides syntax elements (e.g., motion vectors, intra-frame mode indicators, segmentation information, and other such syntax information) to entropy coding unit 56.

[0055] To select a suitable intra-predictive coding mode for the current video block, the intra-predictive processing unit 46 within the prediction processing unit 41 can perform intra-predictive coding of the current video block in relation to one or more neighboring blocks in the same frame as the current block to be encoded to provide spatial prediction. The motion estimation unit 42 and motion compensation unit 44 within the prediction processing unit 41 perform inter-predictive coding of the current video block in relation to one or more prediction blocks in one or more reference frames to provide temporal prediction. The video encoder 20 can execute multiple coding channels, for example, to select a suitable coding mode for each block of video data.

[0056] In some implementations, motion estimation unit 42 determines an inter-frame prediction mode for the current video frame by generating motion vectors based on a predetermined pattern within the video frame sequence. The motion vectors indicate the displacement of a video block within the current video frame relative to a prediction block within a reference video frame. Motion estimation performed by motion estimation unit 42 is the process of generating motion vectors that estimate the motion of video blocks. For example, the motion vectors may indicate the displacement of a video block within the current video frame or picture relative to a prediction block within a reference frame associated with the current block being encoded in the current frame. The predetermined pattern may designate video frames in the sequence as P-frames or B-frames. Intra-frame BC unit 48 may determine vectors (e.g., block vectors) for intra-frame BC coding in a similar manner to how motion estimation unit 42 determines motion vectors for inter-frame prediction, or the block vectors may be determined using motion estimation unit 42.

[0057] Regarding pixel differences, the predicted block for a video block can be, or can correspond to, a block or reference block of a reference frame considered to closely match the video block to be encoded. Pixel differences can be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics. In some implementations, the video encoder 20 can compute values ​​for sub-integer pixel positions of the reference frame stored in the DPB 64. For example, the video encoder 20 can interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference frame. Therefore, the motion estimation unit 42 can perform motion search relative to full-pixel positions and fractional pixel positions and output a motion vector with fractional pixel accuracy.

[0058] The motion estimation unit 42 calculates the motion vector for a video block in an inter-frame predictive coding frame by comparing the position of the video block with the position of the predicted block in a reference frame selected from either a first reference frame list (list 0) or a second reference frame list (list 1), each of the first and second reference frame lists identifying one or more reference frames stored in the DPB 64. The motion estimation unit 42 sends the calculated motion vector to the motion compensation unit 44, and then to the entropy coding unit 56.

[0059] Motion compensation performed by motion compensation unit 44 may involve acquiring or generating prediction blocks based on motion vectors determined by motion estimation unit 42. Upon receiving motion vectors for the current video block, motion compensation unit 44 may locate the prediction block pointed to by the motion vector in a reference frame list within a reference frame list, retrieve the prediction block from DPB 64, and forward the prediction block to adder 50. Adder 50 then forms a residual video block of pixel differences by subtracting the pixel values ​​of the prediction block provided by motion compensation unit 44 from the pixel values ​​of the currently encoded video block. The pixel differences forming the residual video block may include luminance component differences or chrominance component differences, or both. Motion compensation unit 44 may also generate syntax elements associated with video blocks of a video frame for use by video decoder 30 when decoding video blocks of a video frame. Syntax elements may include, for example, syntax elements defining motion vectors for identifying prediction blocks, any flags indicating prediction modes, or any other syntax information described herein. It should be noted that motion estimation unit 42 and motion compensation unit 44 may be highly integrated, but are described separately for conceptual purposes.

[0060] In some implementations, the intra-BC unit 48 can generate vectors and acquire prediction blocks in a manner similar to that described above in conjunction with the motion estimation unit 42 and the motion compensation unit 44; however, these prediction blocks are in the same frame as the current block being encoded, and these vectors are referred to as block vectors rather than motion vectors. Specifically, the intra-BC unit 48 can determine the intra-prediction mode to be used for encoding the current block. In some examples, the intra-BC unit 48 can, for example, use various intra-prediction modes to encode the current block during a separate encoding pass and test their performance through rate-distortion analysis. Next, the intra-BC unit 48 can select a suitable intra-prediction mode from the various tested intra-prediction modes to use and generate an intra-mode indicator accordingly. For example, the intra-BC unit 48 can use rate-distortion analysis to calculate rate-distortion values ​​for the various tested intra-prediction modes and select the intra-prediction mode with the best rate-distortion characteristics from the tested modes as the suitable intra-prediction mode to use. Rate-distortion analysis generally determines the amount of distortion (or error) between the coded block and the original uncoded block that was encoded to produce the coded block, as well as the bit rate (i.e., the number of bits) used to produce the coded block. Intra-frame BC unit 48 can calculate the ratio based on the distortion and rate for various coded blocks to determine which intra-frame prediction mode exhibits the optimal rate-distortion value for the block.

[0061] In other examples, the intra-frame BC unit 48 may use, in whole or in part, the motion estimation unit 42 and the motion compensation unit 44 to perform such functions for intra-frame BC prediction according to the embodiments described herein. In any case, for intra-frame block copying, in terms of pixel differences, the predicted block may be a block considered to closely match the block to be encoded, the pixel differences may be determined by SAD, SSD, or other difference metrics, and identifying the predicted block may include calculating values ​​for sub-integer pixel positions.

[0062] Regardless of whether the predicted block comes from the same frame predicted intra-frame or from a different frame predicted inter-frame, the video encoder 20 can form a residual video block by subtracting the pixel value of the predicted block from the pixel value of the current video block being encoded. The pixel difference forming the residual video block can include both luma component difference and chroma component difference.

[0063] As an alternative to the inter-frame prediction performed by the motion estimation unit 42 and the motion compensation unit 44 as described above, or the intra-block copy prediction performed by the intra-BC unit 48, the intra-prediction processing unit 46 can perform intra-frame prediction on the current video block. Specifically, the intra-prediction processing unit 46 can determine an intra-prediction mode for encoding the current block. To this end, the intra-prediction processing unit 46 can, for example, use various intra-prediction modes to encode the current block during a separate encoding pass, and the intra-prediction processing unit 46 (or, in some examples, a mode selection unit) can select a suitable intra-prediction mode from the tested intra-prediction modes for use. The intra-prediction processing unit 46 can provide information indicating the intra-prediction mode selected for the block to the entropy coding unit 56. The entropy coding unit 56 can encode the information indicating the selected intra-prediction mode into the bitstream.

[0064] After prediction processing unit 41 determines the prediction block for the current video block via inter-frame prediction or intra-frame prediction, adder 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more TUs and provided to transform processing unit 52. Transform processing unit 52 uses a transform (e.g., discrete cosine transform (DCT) or a conceptually similar transform) to transform the residual video data into residual transform coefficients.

[0065] The transform processing unit 52 can send the resulting transform coefficients to the quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process can also reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting the quantization parameters. In some examples, the quantization unit 54 can subsequently perform a scan on the matrix including the quantized transform coefficients. Alternatively, the entropy coding unit 56 can perform the scan.

[0066] After quantization, the entropy coding unit 56 entropy codes the quantized transform coefficients into a video bitstream using, for example, context-adaptive variable-length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probabilistic interval segmented entropy (PIPE) coding, or another entropy coding method or technique. The encoded bitstream can then be sent to, for example,... Figure 1 The video decoder 30 shown, or archived in, for example Figure 1 The data is stored in storage device 32 for later transmission to or retrieval by video decoder 30. Entropy coding unit 56 can also entropy code the motion vectors and other syntax elements used for the current video frame being encoded.

[0067] The inverse quantization unit 58 and the inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct residual video blocks in the pixel domain for generating reference blocks to predict other video blocks. As noted above, the motion compensation unit 44 can generate motion-compensated prediction blocks from one or more reference blocks of frames stored in the DPB 64. The motion compensation unit 44 can also apply one or more interpolation filters to the prediction blocks to compute sub-integer pixel values ​​for use in motion estimation.

[0068] Adder 62 adds the reconstructed residual block to the motion-compensated prediction block generated by motion compensation unit 44 to generate a reference block for storage in DPB 64. The reference block can then be used as a prediction block by intra-frame BC unit 48, motion estimation unit 42, and motion compensation unit 44 for inter-frame prediction of another video block in subsequent video frames.

[0069] Figure 3 This is a block diagram illustrating an exemplary video decoder 30 according to some embodiments of this application. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra-frame prediction unit 84, and an intra-frame BC unit 85. The video decoder 30 can perform operations in conjunction with the above. Figure 2 The decoding process described for the video encoder 20 is essentially the inverse of the encoding process. For example, the motion compensation unit 82 can generate prediction data based on the motion vectors received from the entropy decoding unit 80, while the intra-frame prediction unit 84 can generate prediction data based on the intra-frame prediction mode indicator received from the entropy decoding unit 80.

[0070] In some examples, units of the video decoder 30 may be assigned tasks to perform embodiments of this application. Furthermore, in some examples, embodiments of this disclosure may be distributed across one or more units of the video decoder 30. For example, the intra-frame BC unit 85 may perform embodiments of this application individually or in combination with other units of the video decoder 30 (e.g., motion compensation unit 82, intra-frame prediction unit 84, and entropy decoding unit 80). In some examples, the video decoder 30 may not include the intra-frame BC unit 85, and the functionality of the intra-frame BC unit 85 may be performed by other components of the prediction processing unit 81 (e.g., motion compensation unit 82).

[0071] Video data memory 79 can store video data, such as encoded video bitstreams, that will be decoded by other components of video decoder 30. The video data stored in video data memory 79 can be obtained, for example, from storage device 32, from a local video source (e.g., a camera), via wired or wireless network communication of video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk). Video data memory 79 may include an encoded picture buffer (CPB) that stores encoded video data from the encoded video bitstream. The DPB 92 of video decoder 30 stores reference video data for use by video decoder 30 (e.g., in intra-frame or inter-frame predictive coding modes) when decoding video data. Video data memory 79 and DPB 92 can be formed of any memory device from a variety of memory devices, such as dynamic random access memory (DRAM) (including synchronous DRAM (SDRAM)), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. For illustrative purposes, video data memory 79 and DPB 92 are shown in... Figure 3 The video data memory 79 and DPB 92 are depicted as two distinct components of the video decoder 30. However, it will be apparent to those skilled in the art that the video data memory 79 and DPB 92 may be provided by the same memory device or separate memory devices. In some examples, the video data memory 79 may be on-chip along with other components of the video decoder 30, or off-chip relative to those components.

[0072] During the decoding process, the video decoder 30 receives an encoded video bitstream representing video blocks of encoded video frames and associated syntax elements. The video decoder 30 may receive syntax elements at the video frame level and / or the video block level. The entropy decoding unit 80 of the video decoder 30 performs entropy decoding on the bitstream to generate quantization coefficients, motion vectors or intra-prediction mode indicators, and other syntax elements. The entropy decoding unit 80 then forwards the motion vectors or intra-prediction mode indicators, and other syntax elements to the prediction processing unit 81.

[0073] When a video frame is encoded as an intra-predictive coded (I) frame or used as an intra-coded prediction block in other types of frames, the intra-predictive unit 84 of the prediction processing unit 81 can generate prediction data for the video block of the current video frame based on the intra-predictive mode transmitted by the signal and reference data from the previous decoded block of the current frame.

[0074] When a video frame is encoded as an inter-frame predictive coded (i.e., B or P) frame, the motion compensation unit 82 of the prediction processing unit 81 generates one or more prediction blocks for the current video frame based on motion vectors and other syntax elements received from the entropy decoding unit 80. Each of the prediction blocks can be generated from a reference frame within a reference frame list. The video decoder 30 can construct the reference frame list, i.e., list 0 and list 1, based on the reference frames stored in the DPB 92 using a default construction technique.

[0075] In some examples, when a video block is encoded according to the intra-frame BC mode described herein, the intra-frame BC unit 85 of the prediction processing unit 81 generates a prediction block for the current video block based on the block vector and other syntax elements received from the entropy decoding unit 80. The prediction block can be located within a reconstructed region of the same image as the current video block, as defined by the video encoder 20.

[0076] Motion compensation unit 82 and / or intra-frame prediction (BC) unit 85 determine prediction information for video blocks in the current video frame by parsing motion vectors and other syntax elements, and then use this prediction information to generate prediction blocks for the current video block being decoded. For example, motion compensation unit 82 uses some of the received syntax elements to determine the prediction mode (e.g., intra-frame prediction or inter-frame prediction) used to encode video blocks in the video frame, the inter-frame prediction frame type (e.g., B or P), construction information for one or more reference frames in the reference frame list for the frame, motion vectors for each inter-frame prediction encoded video block in the frame, the inter-frame prediction state for each inter-frame prediction encoded video block in the frame, and other information for decoding video blocks in the current video frame.

[0077] Similarly, the intra-BC unit 85 can use some of the received syntax elements, such as flags, to determine which video blocks in the frame are predicted using the intra-BC mode, which video blocks in the frame are in the reconstruction region and should be stored in the DPB 92, the block vector for each intra-BC predicted video block in the frame, the intra-BC prediction state for each intra-BC predicted video block in the frame, and other information for decoding video blocks in the current video frame.

[0078] The motion compensation unit 82 can also perform interpolation using interpolation filters, such as those used by the video encoder 20 during encoding of video blocks, to calculate interpolated values ​​for sub-integer pixels of the reference block. In this case, the motion compensation unit 82 can determine the interpolation filters used by the video encoder 20 based on the received syntax elements and use these interpolation filters to generate the prediction block.

[0079] The dequantization unit 86 dequantizes the quantized transform coefficients provided in the bitstream and entropy-decoded by the entropy decoding unit 80 using the same quantization parameters calculated by the video encoder 20 for each video block in the video frame to determine the degree of quantization. The inverse transform processing unit 88 applies an inverse transform (e.g., inverse DCT, inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients in order to reconstruct the residual block in the pixel domain.

[0080] After the motion compensation unit 82 or the intra-frame BC unit 85 generates a prediction block for the current video block based on vectors and other syntax elements, the adder 90 reconstructs the decoded video block for the current video block by adding the residual block from the inverse transform processing unit 88 to the corresponding prediction block generated by the motion compensation unit 82 and the intra-frame BC unit 85. A loop filter 91 (e.g., a deblocking filter, SAO filter, CCSAO filter, and / or ALF) may be located between the adder 90 and the DPB 92 for further processing of the decoded video block. In some examples, the loop filter 91 may be omitted, and the decoded video block may be directly provided to the DPB 92 by the adder 90. The decoded video block in a given frame is then stored in the DPB 92, which stores reference frames for subsequent motion compensation of the next video block. The DPB 92 or a separate memory device may also store the decoded video for later presentation on a display device (e.g., ...). Figure 1 On the display device 34).

[0081] In a typical video encoding and decoding process, a video sequence usually consists of an ordered set of frames or images. Each frame may include three sample arrays, denoted as SL, SCb, and SCr. SL is a two-dimensional array of luma samples. SCb is a two-dimensional array of chroma samples (Cb). SCr is a two-dimensional array of chroma samples (Cr). In other instances, a frame may be monochrome and therefore consist of only a two-dimensional array of luma samples.

[0082] like Figure 4A As shown, the video encoder 20 (or more specifically, the segmentation unit 45) generates an encoded representation of a frame by first segmenting the frame into a set of CTUs. A video frame may include an integer number of CTUs ordered consecutively from left to right and top to bottom in raster scan order. Each CTU is the largest logical coding unit, and the width and height of the CTU are signaled by the video encoder 20 in a sequence parameter set such that all CTUs in the video sequence have the same size, one of 128×128, 64×64, 32×32, and 16×16. However, it should be noted that this application is not limited to a specific size. Figure 4BAs shown, each CTU may include a CTB for luma samples, two corresponding coding tree blocks for chroma samples, and syntax elements for encoding the samples of the coding tree blocks. The syntax elements describe the properties of different types of units in the coded pixel block and how the video sequence can be reconstructed at the video decoder 30, including inter-frame prediction or intra-frame prediction, intra-frame prediction mode, motion vectors, and other parameters. In a monochrome image or an image with three separate color planes, the CTU may include a single coding tree block and syntax elements for encoding the samples of that coding tree block. The coding tree block may be an N×N sample block.

[0083] To achieve better performance, the video encoder 20 can recursively perform tree splitting on the coding tree blocks of the CTU, such as binary tree splitting, ternary tree splitting, quadtree splitting, or combinations thereof, and divide the CTU into smaller CUs. For example... Figure 4C As described, the 64×64 CTU 400 is first divided into four smaller CUs, each with a block size of 32×32. Of these four smaller CUs, CU 410 and CU 420 are each divided into four CUs with a block size of 16×16. The two 16×16 CUs, 430 and CU 440, are further divided into four CUs with a block size of 8×8. Figure 4D Depicting as shown Figure 4C The final result of the CTU 400 partitioning process described in the figure is a quadtree data structure, where each leaf node of the quadtree corresponds to a CU of a corresponding size ranging from 32×32 to 8×8. Similar to... Figure 4B The CTU depicted in the image can include, for example, two corresponding coding blocks (CBs) of luminance and chrominance samples of the same frame size, as well as syntax elements for encoding the samples of the coding blocks. In monochrome images or images with three separate color planes, a CU can include a single coding block and a syntax structure for encoding the samples of the coding block. It should be noted that... Figure 4C and Figure 4D The quadtree partitioning depicted is for illustrative purposes only, and a CTU can be split into multiple CUs based on quadtree / ternary / binary partitioning to adapt to varying local characteristics. In multi-type tree structures, a CTU is partitioned according to a quadtree structure, and each quadtree leaf CU can be further partitioned according to binary and ternary tree structures. Figure 4E As shown, a coded block with width W and height H has five possible segmentation types: quad segmentation, horizontal binary segmentation, vertical binary segmentation, horizontal triple segmentation, and vertical triple segmentation.

[0084] In some implementations, the video encoder 20 may further segment the coded blocks of the CU into one or more (M×N) PBs. A PB is a rectangular (square or non-square) block of samples to which the same prediction (inter-frame or intra-frame) is applied. The PU of the CU may include a PB of luma samples, two corresponding PBs of chroma samples, and syntax elements for predicting the PBs. In a monochrome image or an image with three separate color planes, a PU may include a single PB and a syntax structure for predicting the PBs. The video encoder 20 may generate predicted luma blocks, predicted Cb blocks, and predicted Cr blocks for each PU of the CU, representing the luma PB, Cb PB, and Cr PB.

[0085] Video encoder 20 can use intra-frame prediction or inter-frame prediction to generate prediction blocks for the PU. If video encoder 20 uses intra-frame prediction to generate prediction blocks for the PU, then video encoder 20 can generate prediction blocks for the PU based on decoded samples of the frame associated with the PU. If video encoder 20 uses inter-frame prediction to generate prediction blocks for the PU, then video encoder 20 can generate prediction blocks for the PU based on decoded samples of one or more frames other than the frame associated with the PU.

[0086] After the video encoder 20 generates a predicted luminance block, a predicted Cb block, and a predicted Cr block for one or more PUs of the CU, the video encoder 20 can generate a luminance residual block for the CU by subtracting the predicted luminance block of the CU from the original luminance coding block of the CU, such that each sample in the luminance residual block of the CU indicates the difference between a luminance sample in one of the predicted luminance blocks of the CU and a corresponding sample in the original luminance coding block of the CU. Similarly, the video encoder 20 can generate Cb residual blocks and Cr residual blocks for the CU, respectively, such that each sample in the Cb residual block of the CU indicates the difference between a Cb sample in one of the predicted Cb blocks of the CU and a corresponding sample in the original Cb coding block of the CU, and each sample in the Cr residual block of the CU indicates the difference between a Cr sample in one of the predicted Cr blocks of the CU and a corresponding sample in the original Cr coding block of the CU.

[0087] In addition, such as Figure 4CAs shown, the video encoder 20 can use quadtree partitioning to decompose the luminance residual block, Cb residual block, and Cr residual block of the CU into one or more luminance transform blocks, Cb transform blocks, and Cr transform blocks, respectively. A transform block is a rectangular (square or non-square) sample block to which the same transform is applied. A TU of the CU can include a transform block of luminance samples, two corresponding transform blocks of chrominance samples, and syntax elements for transforming the samples of the transform block. Therefore, each TU of the CU can be associated with a luminance transform block, a Cb transform block, and a Cr transform block. In some examples, the luminance transform block associated with a TU can be a sub-block of the CU's luminance residual block. A Cb transform block can be a sub-block of the CU's Cb residual block. A Cr transform block can be a sub-block of the CU's Cr residual block. In a monochrome image or an image with three separate color planes, a TU can include a single transform block and syntax structures for transforming the samples of that transform block.

[0088] The video encoder 20 can apply one or more transforms to the luminance transform block of the TU to generate a luminance coefficient block for the TU. The coefficient block can be a two-dimensional array of transform coefficients. The transform coefficients can be scalars. The video encoder 20 can apply one or more transforms to the Cb transform block of the TU to generate a Cb coefficient block for the TU. The video encoder 20 can apply one or more transforms to the Cr transform block of the TU to generate a Cr coefficient block for the TU.

[0089] After generating coefficient blocks (e.g., luminance coefficient blocks, Cb coefficient blocks, or Cr coefficient blocks), video encoder 20 can quantize the coefficient blocks. Quantization typically refers to the process of quantizing transform coefficients to potentially reduce the amount of data used to represent the transform coefficients, thereby providing further compression. After quantizing the coefficient blocks, video encoder 20 can entropy encode the syntax elements indicating the quantized transform coefficients. For example, video encoder 20 can perform CABAC on the syntax elements indicating the quantized transform coefficients. Finally, video encoder 20 can output a bitstream comprising a bit sequence that forms a representation of coded frames and associated data; the bitstream is stored in storage device 32 or transmitted to target device 14.

[0090] After receiving the bitstream generated by the video encoder 20, the video decoder 30 can parse the bitstream to obtain syntax elements. The video decoder 30 can reconstruct frames of video data, at least in part, based on the syntax elements obtained from the bitstream. The process of reconstructing the video data is generally the inverse of the encoding process performed by the video encoder 20. For example, the video decoder 30 can perform an inverse transform on the coefficient block associated with the TU of the current CU to reconstruct the residual block associated with the TU of the current CU. The video decoder 30 also reconstructs the coded blocks of the current CU by adding the samples of the prediction block of the PU for the current CU to the corresponding samples of the transform block of the TU of the current CU. After reconstructing the coded blocks for each CU of the frame, the video decoder 30 can reconstruct the frame.

[0091] As mentioned above, video coding primarily uses two modes (i.e., intra-frame prediction and inter-frame prediction) to achieve video compression. It should be noted that IBC can be considered either intra-frame prediction or a third mode. Between the two modes, inter-frame prediction contributes more to coding efficiency than intra-frame prediction because it uses motion vectors to predict the current video block based on a reference video block.

[0092] However, with continuously improving video data capture technologies and finer video tile sizes used to preserve details in video data, the amount of data required to represent the motion vectors for the current frame has also increased significantly. One way to overcome this challenge benefits from the fact that not only do a set of neighboring CUs in both the spatial and temporal domains have similar video data for prediction purposes, but the motion vectors between these neighboring CUs are also similar. Therefore, the motion information of spatially neighboring CUs and / or temporally co-located CUs can be used as an approximation of the motion information (e.g., motion vectors) of the current CU (also known as the "Motion Vector Prediction" (MVP) of the current CU) by exploring their spatial and temporal correlations.

[0093] Instead of the above combination Figure 2 The method described involves encoding the actual motion vector of the current CU, determined by the motion estimation unit 42, into the video bitstream, and subtracting the predicted motion vector value of the current CU from the actual motion vector of the current CU to produce the motion vector difference (MVD) for the current CU. By doing so, it is not necessary to encode the motion vector determined by the motion estimation unit 42 for each CU of the frame into the video bitstream, and the amount of data used to represent motion information in the video bitstream can be significantly reduced.

[0094] Similar to the process of selecting a prediction block in a reference frame during inter-frame prediction of a coded block, both the video encoder 20 and the video decoder 30 need to employ a set of rules to construct a motion vector candidate list (also known as a "merging list") for the current CU using those potential candidate motion vectors associated with spatially neighboring CUs and / or temporally co-located CUs. Then, a member is selected from the motion vector candidate list as the motion vector prediction value for the current CU. By doing so, it is not necessary to send the motion vector candidate list itself from the video encoder 20 to the video decoder 30, and the index of the selected motion vector prediction value within the motion vector candidate list is sufficient for both the video encoder 20 and the video decoder 30 to encode and decode the current CU using the same motion vector prediction value from the motion vector candidate list.

[0095] Typically, the basic intra prediction scheme used in VVC is almost identical to that in HEVC, with the difference being the further extension, addition, and / or improvement of several prediction tools, such as extended intra prediction using wide-angle intra mode, multi-reference line (MRL) intra prediction, PDPC, intra-segmentation sub-partition (ISP) prediction, CCLM prediction, and MIP.

[0096] Extended intra-prediction using wide-angle intra-mode Similar to HEVC, VVC uses a set of reference samples that are adjacent to the current CU (i.e., above or to the left of the current CU) to predict the samples of the current CU. However, in order to capture the finer edge orientations present in natural video (especially for high-resolution (e.g., 4K) video content), the number of intra-angular modes is expanded from 33 in HEVC to 93 in VVC. Figure 5 The diagram illustrates a simplified representation of intra-frame modes as defined in VVC. Figure 5 As shown, among the 93 intra-angle modes, modes 2 to 66 are traditional intra-angle modes, while modes -1 to -14 and modes 67 to 80 are wide-angle intra-angle modes. Besides the intra-angle modes, HEVC's planar modes ( Figure 5 Mode 0) and DC mode ( Figure 5 Pattern 1) in VVC is also applied.

[0097] Because VVC uses a quadtree / binary / ternary tree partitioning structure, rectangular video blocks exist in addition to square video blocks for intra-frame prediction in VVC. Since the width and height of a given video block are not equal, different sets of angular intra-frame modes can be selected from 93 angular intra-frame modes for different block shapes. More specifically, for both square and rectangular video blocks, in addition to planar and DC modes, each block shape supports 65 of the 93 angular intra-frame modes. When the rectangular block shape of a video block meets specific conditions, the video decoder 30 can adaptively determine the index of the wide-angle intra-frame mode of the video block based on the index of the conventional angular intra-frame modes received from the video encoder 20, using the mapping relationship shown in Table 1 below. In other words, for non-square blocks, the video encoder 20 uses the index of the conventional angular intra-frame mode to signal the wide-angle intra-frame mode. This index, after being parsed, is mapped by the video decoder 30 to the wide-angle intra-frame mode index, thus ensuring that the total number of intra-frame modes (i.e., planar mode, DC mode, and 65 of the 93 angular intra-frame modes) (i.e., 67) remains unchanged, and the intra-frame mode encoding / decoding method remains unchanged. Therefore, good efficiency in signaling intra-frame modes is achieved while providing a consistent design across different block sizes.

[0098] Table 1 shows the mapping between the indices of the conventional angle intra-mode and the wide angle intra-mode for intra-prediction of different block shapes in VCC, where W represents the width of the video block and H represents the height of the video block.

[0099]

[0100] Table 1 MRL Intra-frame Prediction Similar to intra-prediction in HEVC, all intra-modes in VVC (i.e., planar intra-mode, DC intra-mode, and angular intra-mode) utilize a set of reference samples above and to the left of the current video block for intra-prediction. However, compared to using only the nearest row / column of reference samples (i.e., ... Figure 6 Unlike HEVC (row 0, 601), VVC introduces MRL intra-frame prediction, where, in addition to the nearest row / column of the reference sample, the other two rows / columns of the reference sample (i.e., Figure 6 The first row (603) and the third row (605) are used for intra-frame prediction. The index of the selected row / column of the reference sample is transmitted from the video encoder 20 to the video decoder 30 via signal transmission. When the non-nearest row / column of the reference sample is selected (i.e., Figure 6When the first row (603) or third row (605) is specified, planar modes are excluded from the set of intra-modes that can be used to predict the current video block. MRL intra-prediction is disabled for the first row / column of the video block within the current CTU to prevent the use of extended reference samples outside the current CTU.

[0101] PDPC As mentioned earlier, intra-frame prediction samples are generated from a set of neighboring reference samples, which can introduce discontinuities at block boundaries between the current video block and its neighboring video blocks. VVC introduces the PDPC tool to address this issue by employing a weighted combination of intra-frame prediction samples and boundary reference samples. In VVC, PDPC can be enabled for the following intra-frame modes without transmitting signals: planar mode, DC mode, angular intra-frame modes with indices less than or equal to the horizontal intra-frame mode (i.e., mode 18), and angular intra-frame modes with indices greater than or equal to the vertical intra-frame mode (i.e., mode 50) and less than or equal to 80. PDPC is not applied if the current block is in Block Differential Pulse Code Modulation (BDPCM) mode or if the selected row / column index of the MRL intra-frame prediction reference sample is greater than 0. Assuming the coordinates are ( x , y The predicted sample point for the current sample point at position ) is pred ( x , y Modified prediction samples after PDPC execution pred' ( x , y It is calculated as:

[0102] in, Bitdepth Indicates the bit depth of the sample point. R x',-1 and R -1,y' These represent reference points located at the top and left boundaries of the current sample point, respectively. wL and wT The weights are adaptively selected based on the intra-frame mode and block size of the current block. > > " indicates a bitwise right shift operation, and " << "Indicates bitwise left shift operation."

[0103] The function in equation (1) Clip3(x, y, z) It can be defined as follows:

[0104] Figure 7A and Figure 7BThe diagrams illustrate simplified reference samples for PDPC in the upper right diagonal mode and the lower left diagonal mode, respectively. Predicted Samples pred(x,y) Located within the prediction block ( x , y ( ) at this location. Reference sample point R x',-1 It has a horizontal coordinate x' = x + y + 1 and a vertical coordinate -1, and a reference sample point. R -1,y' It has a horizontal coordinate of -1 and a vertical coordinate of y' = x + y + 1.

[0105] ISP Forecast ISP prediction is a tool applied to the luma intra-frame prediction mode. This tool divides the luma video block vertically or horizontally into 2 or 4 sub-partitions based on its block size, as shown in Table 2. For example, the minimum block size for ISP is 4 × 8 or 8 × 4. Figure 8A and Figure 8B A simplified diagram of sub-partitions depending on the block size is shown. If a video block (e.g., as...) Figure 8A If the block size W × H of the video block 401 shown is equal to 4 × 8 or 8 × 4, then the video block is divided into 2 sub-partitions. If the video block (e.g., as shown) Figure 8B If the block size W × H of the video block 403 shown is greater than 4 × 8 or 8 × 4, then the video block is divided into 4 sub-partitions. The CU size of the ISP is limited to a maximum of 64 × 64. All sub-partitions must satisfy the condition of having at least 16 samples.

[0106]

[0107] Table 2 For each sub-partition, reconstructed samples are obtained by adding the residual signal to the prediction signal. Here, the residual signal is generated through processes such as entropy decoding, inverse quantization, and inverse transform. The reconstructed samples from each sub-partition are used to generate the prediction for the next sub-partition. Furthermore, the first sub-partition to be processed is the one containing the top-left samples of the CU, and after processing the first sub-partition, the ISP prediction continues downwards (for example, ...). Figure 8A and Figure 8B (as shown in the horizontal division) or to the right (for example, Figure 8A and Figure 8B (Vertical segmentation as shown). All sub-partitions share the same intra-frame prediction mode.

[0108] CCLM Forecast To reduce cross-component redundancy, a CCLM prediction mode is used in VVC, where chromaticity samples of the CU are predicted using a linear model based on the reconstructed luminance samples rec_L(i,j) of the CU, as shown below:

[0109] in, This represents the predicted chromaticity samples in the CU. Indicates the reconstructed brightness sample points The downsampled reconstructed luminance samples of the CU obtained after downsampling are given, and α and β are linear model parameters derived from at most four neighboring chrominance samples and their corresponding downsampled luminance samples. Assuming the current chrominance block has a size W × H, W' and H' can be obtained as follows: When applying the LM pattern, W' = W, H' = H; When the LM_A mode is applied, W' = W + H; When the LM_L mode is applied, H' = H + W.

[0110] In the LM mode, the upper and left samples of the CU are used together to calculate the linear model coefficients; in the LM_A mode, only the upper samples of the CU are used to calculate the linear model coefficients; and in the LM_L mode, only the left samples of the CU are used to calculate the linear model coefficients.

[0111] If the position of the upper sample point of the chroma block is represented as S[0, -1]…S[W' - 1, -1] and the position of the left sample point of the chroma block is represented as S[-1, 0]…S[-1, H' - 1], then the positions of the four adjacent chroma samples are selected as follows: - When applying LM mode and both the top and left samples are available, select S[W' / 4, -1]. S[-1, H' / 4] and As the positions of these four neighboring chromaticity sample points; - When applying LM_A mode and the top sample is available, or when only the top sample is available, select As the positions of these four neighboring chromaticity sample points; - When applying LM_L mode and the left-side sample is available, or when only the left-side sample is available, select This refers to the positions of these four neighboring chromaticity sample points.

[0112] The downsampling operation obtains four neighboring brightness samples corresponding to the selected location, and the four neighboring brightness samples are compared four times to find the two larger values: x 0 A and x 1 A and two smaller values: x0 B and x 1 B The chromaticity sample values ​​corresponding to these two larger values ​​and these two smaller values ​​are respectively represented as... y 0 A , y 1 A , y 0 B and y 1 B .Then X a , X b , Y a and Y b It is deduced as:

[0113] Finally, the linear model parameters are obtained according to the following equation. and .

[0114]

[0115] Figure 9 A simplified diagram showing the positions of the left and top samples of the CU involved in the CCLM mode is shown, including the positions of the left and top samples of the N×N chromaticity block 901 in the CU, and the positions of the left and top samples of the 2N×2N luminance block 903 in the CU.

[0116] The above parameter calculations are performed as part of the decoding process, therefore no syntax elements are used to transmit the values ​​of α and β from the video encoder 20 to the video decoder 30.

[0117] MIP MIP is a new intra-frame prediction method added to VVC. In the MIP prediction method, the input consists of H reconstructed neighbor boundary samples from the left side of a rectangular block with width W and height H, and W reconstructed neighbor boundary samples from the top row of the rectangular block. Based on... Figure 10 The following three steps—averaging, matrix-vector multiplication, and linear interpolation—are used to generate the predicted signal for the sample points of the rectangular block.

[0118] First step: Calculate the average of neighboring sample points By sampling adjacent boundary points based on block size and shape bdry top and bdry left The average is used to determine four or eight sample points. Specifically, this is done by averaging the sample points near the boundary according to predefined rules that depend on the block size. bdry top and bdry left Calculate the average and take the samples from the nearest boundary points. bdry top and bdry left Simplified to boundary samples and Then, the simplified boundary samples. and Concatenate to form a simplified boundary vector Therefore, for shapes of The size of the block is 4, while the size of all other block shapes is 8. If If it refers to MIP mode, then this concatenation is defined as follows:

[0119] Second step: Matrix-vector multiplication Using simplified boundary vectors The average samples in the original block are used as input to perform matrix-vector multiplication, and then an offset is added to generate a simplified prediction signal from a set of downsampled samples in the original block. More specifically, the simplified prediction signal... The calculation is as follows:

[0120] here, It is a matrix, and the matrix is ​​in Time has It has 4 rows and 4 columns, or 8 columns in all other cases. It is the size of The offset vector.

[0121] here, and Defined as:

[0122] matrix and offset vector Taken from set , , One of them. From this, a matrix is ​​derived. and offset vector index of the set Defined as follows:

[0123] Here, matrix Each coefficient is represented with 8 bits of precision. (Set) Composed of 16 matrices and 16 offset vectors The structure consists of matrices with 16 rows and 4 columns, and each offset vector has a size of 16. The matrix and offset vector of this set are used for a size of The block. Set Composed of 8 matrices and 8 offset vectors The structure, in which each matrix has It has 8 rows and 8 columns, and each offset vector has a size of 16. .gather Composed of 6 matrices and 6 offset vectors The structure consists of matrices with 64 rows and 8 columns, and each offset vector has a size of 64. .

[0124] Third step: Interpolation The predicted signal at the remaining locations is generated from the simplified predicted signal of the downsampled set of samples through linear interpolation, which is a single-step linear interpolation in each direction. Regardless of the block shape or size, the interpolation is performed first in the horizontal direction and then in the vertical direction.

[0125] Matrix-based position-related intra-frame prediction replaces existing traditional intra-frame prediction (PDP). A weight matrix defined for block shape and intra-frame mode is introduced. These weights are multiplied by a neighboring reference template to obtain predicted samples, thus replacing traditional intra-frame prediction. The weights are applied to the reference samples of an L-shaped causal neighborhood template, such as... Figure 11 As shown.

[0126] The reference sample in the causal neighborhood is denoted as r, and F(x,y) is the weight matrix. The prediction P(x,y) can then be derived as...

[0127] Where k represents the index of the reference sample point in the template.

[0128] In testing, this prediction was applied to block sizes with both width and height not exceeding 32 (except for 4 × 32, 32 × 4, 8 × 32, and 32 × 8). For blocks with both width and height not exceeding 16, the template size was 2, and it was only used for modes 0, 1, and 2. For other blocks, the template size is set to 1; this is used for modes 0, 1, and... Prediction is performed only for 16 × 16 locations, with the remaining samples generated via bilinear interpolation. Symmetry based on block shape and pattern is used for all block sizes. For patterns greater than 18 and less than 50, the reference length is set to W and H, while in other cases it is set to... .

[0129] Modification of matrix-based position-related intra-frame prediction The matrix is ​​retrained, and the size of the prediction block generated by matrix multiplication is equal to the size of the current block, without subsampling.

[0130] Intra-frame prediction based on neural networks in NNVC Neural network inference Intra-frame prediction modes based on neural networks include There are 1 neural network, each prediction block has 1 Different sizes are available. The size is... The neural network prediction block is represented as ,in, It contains its parameters. For a given piece , Get the block located above it. OK A reference sample row and the left side of the block. List The context composed of reference samples Preprocessed version to provide .right Apply post-processing to produce Prediction See Figure 12 .also, Return two indexes and . This represents the index that characterizes the LFNST kernel index, and indicates when... , When applying DCT-2 to the residuals predicted by the neural network in both the horizontal and vertical directions, whether the primary transform coefficients are transposed, see [reference needed]. Figure 12 .also, An index is provided for VVC intra-prediction modes (PLANAR intra-prediction mode, DC intra-prediction mode, or directional intra-prediction mode). This model is based on Surrounding reference sample pairs The prediction best represents See Figure 12 .

[0131] if :

[0132] otherwise: if

[0133]

[0134] otherwise:

[0135] if

[0136]

[0137] otherwise:

[0138] if ,but .otherwise, .

[0139] if ,but .otherwise, .

[0140] Preprocessing and postprocessing Preprocessing of the current block's context Figure 12 The “preprocessing” shown includes the following four steps.

[0141] See Figure 13 ,from Subtract Reference points available in China average .

[0142] If the neural network predicting the current block uses floating-point numbers, then the context... The reference sample points in the middle need to be multiplied by , The internal bit depth, i.e., in VVC. Otherwise, the context The reference sample points in the middle need to be multiplied by , This indicates the input quantizer.

[0143] All unavailable reference points (See) Figure 13 All are set to .

[0144] The context obtained from the previous step is flattened, resulting in a size of vector .

[0145] Post-processing of neural network predictions Figure 12 The "post-processing" described in the text includes resizing the data to a size of... vector Plastic surgery to height And the width is The rectangle, the result of the integer transformation is divided by The average value of the reference samples available in the context of the current block. Addition, and extraction Therefore, post-processing can be summarized as follows:

[0146] Adaptive derivation of the MPM list When creating a list of MPMs for a given luminance CB, if the "left" luminance CB is predicted via a neural network-based intra-frame prediction mode, the neural network-based mode index can be returned during the prediction of the "left" luminance CB. The index is then replaced and becomes a candidate index to be added to the MPM list. Similarly, if the "above" luminance CB is predicted via a neural network-based intra-frame prediction mode, the neural network-based mode index can be returned during the prediction of the "above" luminance CB. Replace it and become a candidate index to be inserted into the MPM list.

[0147] Signaling based on neural network-based intra-frame prediction mode Signaling based on neural network-based intra-frame prediction mode in luminance The position of its top-left pixel in the current luminance channel The current situation In the luminance CB, intra-predictive mode signaling in luminance is divided into two cases.

[0148] if ,but nnFlag It appears in the intra-prediction mode signaling in the brightness. nnFlag This means selecting a neural network-based intra-frame prediction mode to predict the current brightness CB and then ending (END). nnFlag This means that a neural network-based intra-prediction mode was not selected to predict the current luminance CB, and then the regular intra-prediction mode signaling in the luminance was applied (represented as...). See also Figure 14 .

[0149] Otherwise, apply regular intra-frame prediction mode signaling in the lumen. .

[0150] Note that in " In the case of "", if the context of the current luminance CB exceeds the boundary of the current luminance channel, i.e. In this case, intra-frame prediction based on neural networks is replaced by PLANA.

[0151]

[0152] Signaling for intra-frame prediction modes based on neural networks in chroma The top-left pixel is located at the center of the current chroma channel pair. The current situation For chroma CB pairs, intra-frame prediction mode signaling in chroma is divided into two cases.

[0153] If the luminance CB, which is in the same position as this pair of chrominance CB, is predicted using an intra-frame prediction mode based on a neural network: if Then DM becomes an intra-frame prediction mode based on neural networks.

[0154] Otherwise, set DM to PLANA.

[0155] otherwise: if Then nnFlagChroma appears in the intra-prediction mode signaling in chroma. nnFlagChroma is placed before the DM flag in the decision tree of the intra-prediction mode signaling in chroma. This means selecting a neural network-based intra-frame prediction mode to predict the current chroma C / B pair and then ending the process. nnFlagChroma This means that a neural network-based intra-prediction mode was not selected to predict the current chroma CB pair, and then the regular intra-prediction mode signaling in the chroma was recovered from the DM flag.

[0156] Otherwise, apply the standard intra-frame prediction mode signaling in chroma.

[0157] Note that in " "in the case of" and " In the case of "", if the context of the current chroma CB exceeds the boundary of the current chroma channel, i.e. In this case, intra-frame prediction based on neural networks is replaced by PLANA.

[0158] Transformation of context and neural network prediction For a given block, if Therefore, a neural network-based intra-frame prediction model might have to predict this block, but a neural network-based intra-frame prediction model does not include... In this situation, Figure 12 Before the step known as "preprocessing," the context of the current block can be downsampled in the vertical direction. Multiply and / or downsample in the horizontal direction Doubled and / or transposed. Then, in Figure 12 Following a step known as "post-processing," the prediction for the current block can be transposed and / or upsampled in the vertical direction. Multiply and / or upsample in the horizontal direction Multiple. Select the context of the current block and the transpose of the prediction. and This allows the use of neural networks belonging to the intra-frame prediction mode based on neural networks for prediction, as shown in Table 3.

[0159]

[0160] Table 3: For each For the current situation to be predicted The context of the block and the transpose of the prediction for that block, The value and The value of , and the decision of the neural network used for prediction, which belongs to the intra-frame prediction mode based on neural networks.

[0161] High-complexity and low-complexity versions of neural network-based intra-prediction modes

[0162]

[0163] Intra-frame prediction based on neural networks used in ECM The intra-frame prediction algorithm and model based on neural networks are described below: SADL is used for reasoning. Model weights and intermediate results use 16-bit integers, while internal calculations use 32-bit integers. There are 7 models, depending on the block size. Sparse weights are enforced during training to reduce complexity. SADL supports simple sparse matrix storage (row compression method) and sparse matrix multiplication algorithms, thus leveraging matrix sparsity. The sparsity constraints are strengthened compared to JVET-AH0156.

[0164] Table 4 shows the parameter count and storage memory required for each model. The reported memory usage is for reference only, as it depends on the implementation of the sparse matrix storage. Table 4 reports the total memory storage for the actual tested ECM implementation.

[0165]

[0166] Table 4: Parameter count and storage memory required for each model.

[0167] Table 5 reports the complexity of different neural network models, measured in MAC / pixel. The bottom of the table shows the complexity of the blocks predicted using downsampled reference samples.

[0168]

[0169] Table 5: Complexity of different neural network models as measured in MAC / pixel.

[0170] Intra-frame prediction based on neural networks derived using DIMD modes The proposed neural network-based intra-frame prediction model is described as follows: Using dense matrix multiplication Apply modified linear activation ReLU instead of leakage ReLU Using DIMD to derive intra-frame modes, instead of a prediction network SADL is used for reasoning. Seven models were used, with parameters quantized as 16-bit integers (as in the NNVC method). Table 6 reports the complexity of different neural network models, measured in MAC / pixel. The bottom of the table shows the complexity of the blocks predicted using downsampled reference samples.

[0171]

[0172] Table 6: Complexity of different neural network models as measured in MAC / pixel.

[0173] GPM In VVC, GPM is supported for inter-frame prediction. GPM is signaled using CU-level flags as one merging mode; other merging modes include regular merging, MMVD, CIIP, and sub-block merging. For each possible CU size... ( in, (excluding 8) 64 and 64 8) GPM supports a total of 64 partitions.

[0174] When using GPM, the CU is divided into two parts by a geometrically positioned straight line. The position of the dividing line is mathematically derived based on the angle and offset parameters of the specific partition. Inter-frame prediction is performed using the motion of each part of the CU obtained through geometric partitioning; and each partition allows only unidirectional prediction, i.e., each part has one motion vector and one reference index. Unidirectional prediction motion constraints are applied to ensure that, as with traditional bidirectional prediction, each CU requires only two motion-compensated predictions.

[0175] If GPM is used for the current CU, then the geometric partition index (indicating the angle and offset of the geometric partition) and two merge indexes (one for each partition) are further transmitted using signal transmission.

[0176] The unidirectional prediction candidate list is derived directly from the merging candidate list constructed according to the extended merging prediction process described above. Let n denote the index of the unidirectional prediction motion vector in the unidirectional prediction candidate list. The LX motion vector of the nth merging candidate in the merging candidate list (where X equals the parity of n) is used as the nth unidirectional prediction motion vector for GPM. These motion vectors are... Figure 15 The symbol is marked with "x". If the corresponding LX motion vector of the nth merge candidate in the merge candidate list does not exist, the L(1 - X) motion vector of the same merge candidate is used as the unidirectional predicted motion vector of GPM.

[0177] CIIP In VVC, when encoding and decoding a CU in merge mode, if the CU contains at least 64 luma samples (i.e., the width of the CU multiplied by the height of the CU is equal to or greater than 64) and if both the width and height of the CU are less than 128 luma samples, an additional flag indicating whether CIIP mode is applied to the current CU is transmitted via signaling. In CIIP mode, the prediction signal is obtained by combining the inter-frame prediction signal with the intra-frame prediction signal. The inter-frame prediction signal in CIIP mode is derived using the same inter-frame prediction process applied in regular merge mode; and the intra-frame prediction signal in CIIP mode is derived after utilizing the regular intra-frame prediction process in planar mode. Then, a weighted average is used to combine the intra-frame prediction signal and the inter-frame prediction signal, where, according to (e.g.) Figure 16 (As shown) The weight values ​​are calculated based on the encoding patterns of the top and left neighboring blocks of the current CU 1601 as follows: —If the top neighboring block is available and intra-coded, isIntraTop is set to 1; otherwise, isIntraTop is set to 0. —If the left neighboring block is available and has been intra-coded, isIntraLeft is set to 1; otherwise, isIntraLeft is set to 0. —If (isIntraLeft + isIntraTop) equals 2, then set the weight value to 3; —Otherwise, if (isIntraLeft + isIntraTop) equals 1, then set the weight value to 2; —Otherwise, set the weight value to 1.

[0178] — Prediction signal in CIIP mode The derivation is as follows:

[0179] in, It is the inter-frame prediction signal in CIIP mode. It is an intra-frame prediction signal in CIIP mode. It represents the weight value, and >> indicates a right shift operation.

[0180] Decoder-side intra-frame mode derivation (DIMD) When applying DIMD, up to five intra-frame modes are derived from the reconstructed neighboring samples. These five predictions are then combined with non-directional predictions (based on planar or block vectors), with weights derived from the gradient histogram, as described in JVET-O0449. The decision between non-directional modes is made based on template cost. Specifically, all adjacent and non-adjacent merge candidate block vectors (encoded in IntraTMP or IBC) are compared with planar predictions on the reconstructed template. Template cost (SATD) is used to select the best prediction.

[0181] Division operations in weight derivation are performed using the same lookup table (LUT)-based integerization scheme used in CCLM. For example, division operations in orientation calculation:

[0182] The following LUT-based method is used for calculation:

[0183] in, DivSigTable

[16] = {0, 7, 6, 5,5, 4, 4, 3, 3, 2, 2, 1, 1, 1, 1, 0}.

[0184] For size If the histogram magnitude of the block above or to the left is twice that of the other, then the weight of each of the five derived patterns is modified. In this case, the weights are position-dependent and calculated as follows: If the histogram above is twice the size of the one on the left, then: .

[0185] If the histogram on the left is twice the size of the one above, then: , in, It is the unmodified uniform weight of DIMD as selected in JVET-O0449. It is predefined and set to 10.

[0186] The derived intra-frame modes are included in the main list of most probable intra-frame modes (MPMs), so the DIMD process is performed before the MPM list is built. The main derived intra-frame modes of the DIMD block are stored with the block and used for the construction of the MPM list of neighboring blocks.

[0187] Finally, note that, compared to the JVET-O0449 method, the region used to compute the gradient histogram for neighboring reconstructed samples is modified based on the availability of reconstructed samples. The region of the decoded reference sample for the current W × H brightness CB expands towards the upper right (if available), by a maximum of W additional columns. It expands towards the lower left (if available), by a maximum of H additional rows.

[0188] Template-based intra-frame mode derivation (TIMD) fusion For each intra-prediction mode in the MPM, and for the wide-angle mode where reference samples at the top right and / or bottom left corners are available, the SATD between the predicted and reconstructed samples of the template is calculated. The top two intra-prediction modes with the minimum SATD and one non-angle intra-prediction mode (i.e., DC or plane) with the lowest SATD cost are selected as the TIMD modes. After applying the PDPC procedure, these three TIMD modes are fused with weights, and the current CU is encoded using this weighted intra-prediction. Position-dependent intra-prediction combination (PDPC) is included in the derivation of the TIMD modes.

[0189] Check the following conditions to determine whether to use non-angular intra-frame prediction mode in fusion: The non-angle intra-prediction mode differs from the two selected intra-prediction modes.

[0190] Where costMode3 is the SATD cost of the non-angle intra-prediction mode, and costMode1 is the SATD cost of the first intra-prediction mode.

[0191] If both conditions are met, then three intra-prediction modes are used to generate predictions. The weights of each intra-prediction mode are then calculated based on the SATD cost. .

[0192] Otherwise, non-angular intra-frame prediction modes are not used in the prediction. The costs of the two selected modes are compared to a threshold; in testing, a cost factor of 2 is applied as follows:

[0193] If the condition is met, then apply fusion; otherwise, use only mode 1.

[0194] The weights of the patterns are calculated based on their SATD costs as follows: weight1 = costMode2 / (costMode1+ costMode2) weight2 = 1 - weight1 Division operations are performed using the same lookup table (LUT)-based integerization scheme used by CCLM.

[0195] Furthermore, the location-dependent sample-based fusion used in the DIMD fusion process will be applied to TIMD fusion, but the location-dependent criterion applied to the magnitude of the selected predictions will be replaced by a criterion based on SATD cost. The location-dependent criterion is determined based on the ratio of the normalized SATD of the selected TIMD predictions calculated in the upper template region and the left template region.

[0196] CIIP improvements Sub-block CIIP Sub-block-based merging candidates can be used to generate CIIP inter-frame signals, utilizing the same sub-block-based merging candidate list used by affine and sbTMVP.

[0197] When the CIIP flag is true and the CIIP-TM flag is false, the sub-block-based CIIP flag is signaled. If the sub-block-based CIIP flag is true, the index of a specific candidate in the sub-block-based merge list is signaled, and TIMD is used by default to generate intra-frame signaling, so the CIIP-PDPC flag is no longer signaled.

[0198] CIIP combined with TIMD and TM In CIIP mode, prediction samples are generated by weighting the inter-prediction signals that use CIIP-TM to merge candidate predictions and the intra-prediction signals that use TIMD-derived intra-prediction modes. This method is only applicable to coded blocks with an area of ​​1024 or less.

[0199] The TIMD derivation method is used to derive intra-prediction modes in CIIP. Specifically, the intra-prediction mode with the smallest SATD value is selected from the TIMD mode list and mapped to one of 67 regular intra-prediction modes.

[0200] Furthermore, it is proposed that if the derived intra-prediction mode is an angle mode, the weights of these two tests (wIntra, wInter) should be modified. For near-horizontal mode (2 <= angle mode index < 34), the current block is as follows: Figure 17 (a) shows a vertically divided block; for near-vertical mode (34 <= angle mode index <= 66), the current block is as follows: Figure 17 (b) shows the horizontal division.

[0201] Table 7 shows the different sub-blocks (wIntra, wInter).

[0202]

[0203] Table 7. Modification weights used for angle mode.

[0204] Using CIIP-TM, a CIIP-TM merge candidate list is established for the CIIP-TM pattern. The merge candidates are refined through template matching. The CIIP-TM merge candidates are also reordered into regular merge candidates using the ARMC method. The maximum number of CIIP-TM merge candidates is two.

[0205] Geometric Partitioning (GPM) in ECM GPM using combined motion vector difference (MMVD) The GPM in VVC is extended by applying motion vector refinement on top of the existing GPM unidirectional MV. First, a flag for the GPM CU is transmitted via signal transmission to specify whether a mode is used. If the mode is used, each geometric partition of the GPM CU can further determine whether to use signal transmission MVD. If signal transmission MVD is used for a geometric partition, the motion of the partition is further refined using the signal transmission MVD information after selecting a GPM merging candidate. All other procedures remain the same as in the GPM.

[0206] Similar to MMVD, MVD is transmitted as a pair of range and direction signals. In GPM utilizing MMVD (GPM-MMVD), nine candidate ranges are involved. Pixels The system supports 1 pixel, 2 pixels, 3 pixels, 4 pixels, 6 pixels, 8 pixels, and 16 pixels, and has eight candidate directions (four horizontal / vertical directions and four diagonal directions). Additionally, when pic_fpel_mmvd_enabled_flag equals 1, the MVD is shifted left by 2 bits, as in MMVD.

[0207] GPM using Template Matching (TM) Template matching is applied to GPM. When GPM mode is enabled for CU, a CU-level flag is signaled to indicate whether TM is applied to both geometric partitions. TM is used to refine the motion information for each geometric partition. When TM is selected, a template is constructed using left neighbor samples, top neighbor samples, or both left neighbor samples and top neighbor samples, based on the partition angles shown in Table 8. Motion is then refined by minimizing the difference between the current template and the template in the reference image using the same search mode with the merging mode having the half-pixel interpolation filter disabled.

[0208]

[0209] Table 8. Templates for the first and second geometric partitions, where A indicates the use of the top sample point, L indicates the use of the left sample point, and L+A indicates the use of both the left and top sample points.

[0210] The GPM candidate list is constructed as follows: 1. Derive interleaved list 0 MV candidates and list 1 MV candidates directly from the regular merge candidate list, where list 0 MV candidates have higher priority than list 1 MV candidates. Apply an adaptive threshold pruning method based on the current CU size to remove redundant MV candidates.

[0211] 2. Directly derive interleaved List 1 MV candidates and List 0 MV candidates from the regular merge candidate list, where List 1 MV candidates have higher priority than List 0 MV candidates. The same pruning method using adaptive thresholds is also applied to remove redundant MV candidates.

[0212] 3. Fill the zero MV candidate list until the GPM candidate list is full.

[0213] GPM-MMVD and GPM-TM are enabled only for one GPM CU. This is achieved by first signaling the GPM-MMVD syntax. When both GPM-MMVD control flags are false (i.e., GPM-MMVD is disabled for both GPM partitions), the GPM-TM flag is signaled to indicate whether template matching is applied to both GPM partitions. Otherwise (if at least one GPM-MMVD flag is true), the value of the GPM-TM flag is inferred to be false.

[0214] GPM using inter-frame and intra-frame prediction In GPM utilizing inter-frame and intra-frame prediction, the final prediction samples are generated by weighting the inter-frame and intra-frame prediction samples of each GPM-separated region. Inter-frame prediction samples are obtained from inter-frame GPM, while intra-frame prediction samples are obtained from the intra-frame prediction mode (IPM) candidate list and the index transmitted from the encoder signal. The IPM candidate list size is predefined as 3. Available IPM candidates are the parallel angle mode (parallel mode) for GPM block boundaries, the vertical angle mode (vertical mode) for GPM block boundaries, and others, etc. Figure 18 (a) to Figure 18 (c) shows the planar pattern. Furthermore, as... Figure 18 (d) shows that the use of intra-frame and intra-frame prediction GPM is limited to reduce the signal transmission overhead of IPM and avoid increasing the size of intra-frame prediction circuitry on the hardware decoder. Additionally, direct motion vectors and IPM storage are introduced in the GPM mixing region to further improve encoding and decoding performance.

[0215] In IPM derivation based on DIMD and neighboring modes, parallel modes are registered first. Therefore, if no identical IPM candidates exist in the list, a maximum of two IPM candidates can be registered using the decoder-side intra-frame mode derivation (DIMD) method and / or neighboring block derivation. As for neighboring mode derivation, a maximum of five neighboring block locations are available, but these locations are limited by the GPM block boundary angles (as shown in Table 9 below), which have already been used to utilize template-matched GPM (GPM-TM).

[0216]

[0217] Table 9. Locations of available neighboring blocks derived from IPM candidate derivation based on the angles of the GPM block boundaries. A and L represent the top and left sides of the predicted block.

[0218] GPM-intraframe can be combined with GPM using motion vector difference merging (GPM-MMVD). TIMD is used on IPM candidates within GPM-intraframe to further improve encoding / decoding performance. Parallel modes can be registered first, followed by TIMD, DIMD, and IPM candidates for neighboring blocks.

[0219] Template matching-based GPM segmentation pattern reordering In template-match-based GPM segmentation pattern reordering, given the motion information of the current GPM block, the corresponding TM generation value of the GPM segmentation pattern is calculated. Then, all GPM segmentation patterns are reordered in ascending order based on their TM generation values. Instead of sending GPM segmentation patterns, a signaling method using Golomb-Rice codes is used to indicate the exact index of the GPM segmentation pattern within the reordering list.

[0220] The GPM partition reordering method is a two-step process performed after generating the corresponding reference templates for the two GPM partitions in the coding unit, as follows: • Extend the GPM partition edge to the reference templates of the two GPM partitions to generate 64 reference templates, and compute the corresponding TM cost for each of the 64 reference templates. • The TM generation values ​​based on the GPM splitting pattern are reordered in ascending order, and the top 32 are marked as available splitting patterns.

[0221] like Figure 19 As shown, the edges on the template extend from the edges of the current CU, but the GPM blending process does not apply to the template region on that edge.

[0222] After reordering in ascending order using TM cost, the index is transmitted using semaphores.

[0223] Utilizing an adaptive blending geometric partitioning model (GPM) In VVC, the final predicted samples are generated by mixing the predictions of the two predicted signals using a weighted average. Two integer mixing matrices (W0 and W1) are used. The weights in the GPM mixing matrix are derived from the ramp function based on the displacement from the predicted sample location to the GPM partition boundary. The mixing region size is fixed at two (two samples on each side of the GPM partition boundary).

[0224] The blending process in ECM is improved by adding four additional blending region sizes (one-quarter, half, twice, and four times the existing region size), such as Figure 20As shown, the CU-level flag, encoded by signal transmission, represents the selected mixing region size. Furthermore, extended weighted precision is utilized, where the maximum weighted value changes from 8 (in VVC) to 32 to accommodate the extended mixing region size.

[0225] Two-way prediction of GPM The GPM design in VVC relies on unidirectional predicted motion vectors to generate motion-compensated predicted samples for each inter-frame GPM partition. In ECM, this design has been extended to allow the use of bidirectional predicted motion vectors.

[0226] When constructing the GPM candidate list, the extraction process for extracting unidirectional predicted motion vectors from the initial merge list is invoked only for small blocks of 8 × 8, 16 × 8, and 8 × 16. For larger blocks, the extraction process is bypassed, so the initial merge list (which may contain merged Bi-MVs) is directly used as the final GPM merge list. The generation of the initial merge list is the same as before (i.e., normal merge list generation without any candidate reordering), except that when generating the initial merge list for larger blocks (i.e., blocks that bypass the extraction process), the motion vector difference threshold used to control whether candidates can be added to the list is increased to a full sample distance.

[0227] When generating motion-compensated prediction samples, BDOF-based motion vector refinement, such as in multi-pass DMVR, is used.

[0228] When GPM-MMVD is used for GPM partitioning and its underlying motion vectors are bidirectionally predicted, for low-latency images, MVD with signal transmission is applied to the L0 and L1 motion vectors, as in existing merged MMVD designs. For non-low-latency images, the bidirectional predicted motion vectors are first converted to unidirectional predicted motion vectors, and then MVD is applied to them.

[0229] AMC-GPM In ECM, GPM is further extended to enable affine motion compensation (AMC). Therefore, GPM partitions can be predicted via AMC inter-frame prediction, non-AMC inter-frame prediction, or intra-frame prediction. Furthermore, GPM partitions predicted by AMC can be combined with other GPM partitions predicted via AMC, non-AMC, or intra-frame prediction.

[0230] When AMC is applied, a one-way predictive affine merge candidate list is constructed from the sub-block-based merge candidate list after discarding sub-TMVP candidates, similar to the one-way predictive merge candidate list construction for GPM in VVC. AMC is performed on the GPM partition using the control point motion vectors (CPMVs) of the merge candidates in the one-way predictive affine merge candidate list. The length of the one-way predictive affine merge candidate list is signaled in SPS. When ARMC is applicable, the one-way predictive affine merge candidate list is reordered according to the template cost.

[0231] For each GPM partition, a semaphore `gpm_affine_flag` is used to indicate whether AMC should be applied to the GPM partition. Depending on whether AMC or non-AMC is applied, a separate arithmetic context model is used to semaphore the merge candidate indexes for the GPM partition.

[0232] AMC is not permitted for use with GPM-MMVD and GPM-TM.

[0233] Implicit GPM In implicit GPM, two integer mixing matrices ( W 0 and W 1) Derived from the template (top row, left column). The blending matrix is ​​modeled as an affine ray function of the sample location (x, y) in the current CU: W0(x,y) = ax + by + c and W1(x,y) = 1 - W0(x,y) The parameters (a, b, c) are derived from the reference template using the same solver (MSE minimization) as used for CCCM, GLM, or GL-CCCM. The candidate pair list is constructed from regular GPM candidates and reordered according to the template cost.

[0234] GPM implicit mode via CU-level flags ( gpm_implicit_flag (Transmission via signal.) If gpm_ implicit_flag If true, then for merge-idx Encode the GPM candidate pairs to be used for signal transmission. If gpm_ implicit_flag If false, then use signal transmission for regular GPM syntax elements.

[0235] Spatial Geometric Partitioning Model (SGPM) SGPM is an intra-frame mode similar to GPM's inter-frame coding / decoding tools, where two prediction components are generated based on the intra-frame prediction process. In this mode, a candidate list is built, where each entry contains one partitioning mode and two intra-frame prediction modes, such as... Figure 21As shown. 26 partitioned modes and 3 intra-frame prediction modes are used to form a combination. The candidate list length is set to 16. The selected candidate index is transmitted via signaling.

[0236] Reorder the list using a template. Figure 22 The SAD between the template's prediction and reconstruction is used for sorting. The template size is fixed at 1.

[0237] For each partition mode, the same intra-inter-frame GPM list derivation is used to derive the IPM list for each partition. The IPM list size is set to 3. In this list, the TIMD derivation mode is replaced by two derivation modes with horizontal and vertical orientations. The list is further expanded using block vector-based prediction candidates obtained from adjacent and non-adjacent merge candidates encoded in IntraTMP or IBC modes. Template cost is used to select up to 6 block vectors. The final list contains up to 9 predictions: 3 regular intra-frame modes and up to 6 block vector-based predictions.

[0238] The SGPM mode is applied to constrained block sizes: 4 <= width <= 64, 4 <= height <= 64.

[0239] The PPS flag is encoded to indicate whether blending of two intra-frame predictions is not allowed. When the PPS flag is set to false, the following adaptive blending is also applied to spatial GPM, where, Figure 23 The mixing depth τ shown is derived as follows: If min(width, height) == 4, then choose 1 / 2τ. Otherwise, if min(width, height) == 8, then choose τ. Otherwise, if min(width, height) == 16, then choose 2τ. Otherwise, if min(width, height) == 32, then choose 4τ. Otherwise, choose 8τ Otherwise (with the PPS flag set to true), 1 / 4τ is always used for spatial GPM encoded blocks to ensure that blending is not used when SGPM blocks have fully horizontal or vertical partition angles, and a much narrower blending width is used when SGPM blocks have other partition angles. Note that the flag is set to true in the Current Common Test Conditions (CTC) for Screen Content Video.

[0240] The inventors of this application recognize that currently, intra-prediction signals in CIIP are predicted using intra-prediction modes derived from TIMD. Considering that the prediction accuracy of neural network-based intra-prediction modes is generally superior to that of traditional intra-prediction modes, it is best to incorporate neural network-based intra-prediction modes when deriving intra-prediction signals in CIIP, which can further improve encoding and decoding performance.

[0241] The inventors of this application recognize that currently, intra-prediction signals in GPM are predicted using parallel modes from neighboring blocks, TIMD-derived modes, DIMD-derived modes, or intra-prediction modes. Considering that neural network-based intra-prediction modes generally have better prediction accuracy than traditional intra-prediction modes, it is best to incorporate neural network-based intra-prediction modes when deriving intra-prediction signals in GPM, which can further improve encoding and decoding performance.

[0242] The inventors of this application recognize that currently, intra-prediction signals in SGPM are predicted using parallel modes from neighboring blocks, horizontal TIMD-derived modes, vertical TIMD-derived modes, DIMD-derived modes, or intra-prediction modes. Considering that the prediction accuracy of neural network-based intra-prediction modes is generally superior to that of traditional intra-prediction modes, it is preferable to incorporate neural network-based intra-prediction modes when deriving intra-prediction signals in SGPM, which can further improve encoding and decoding performance.

[0243] The inventors of this application recognize that currently, in TIMD, the SATD (Self-Supporting Targeting) between predicted and reconstructed samples of a template is used to select an intra-prediction mode or fuse several intra-prediction modes. Considering that the prediction accuracy of neural network-based intra-prediction modes is generally better than that of traditional intra-prediction modes, it is preferable to introduce neural network-based intra-prediction modes into TIMD, which can further improve encoding and decoding performance.

[0244] The inventors of this application recognize that currently, in DIMD, gradient histograms are used to select several intra-prediction modes and these modes are fused with their corresponding weights. Considering that the prediction accuracy of neural network-based intra-prediction modes is generally better than that of traditional intra-prediction modes, it is preferable to introduce neural network-based intra-prediction modes into DIMD, which can further improve encoding and decoding performance.

[0245] The inventors of this application recognize that currently, in DIMD, gradient histograms are used to select several intra-prediction modes and they are fused with corresponding weights. Considering that the prediction accuracy of matrix-based position-dependent intra-prediction modes (PDPs) is generally better than that of traditional intra-prediction modes, it is preferable to introduce matrix-based position-dependent intra-prediction modes (PDPs) into DIMD, which can further improve encoding and decoding performance.

[0246] This disclosure provides methods for further improving existing designs of CIIP, GPM, SGPM, TIMD, and DIMD. The main features of the techniques presented in this disclosure are generally summarized below.

[0247] Combining neural network-based intra-frame prediction with CIIP.

[0248] Intra-frame prediction based on neural networks combined with GPM.

[0249] Combination of neural network-based intra-frame prediction and SGPM.

[0250] Combining neural network-based intra-frame prediction with TIMD.

[0251] Intra-frame prediction based on neural networks combined with DIMD.

[0252] Matrix-based position-related intra-frame prediction combined with DIMD.

[0253] It should be noted that the disclosed methods can be applied independently or in combination.

[0254] Neural network-based intra-frame prediction in combination with CIIP According to one or more embodiments of this disclosure, neural network-based intra-frame prediction is combined with CIIP. Different methods can be used to achieve this goal.

[0255] In the first method, the intra-prediction signal predicted using the intra-prediction mode derived using TIMD is replaced by the intra-prediction signal predicted using a neural network-based intra-prediction mode. Here, the intra-prediction mode representation of a block can be an intra-prediction mode derived using TIMD, an intra-prediction mode predicted by a neural network, or an intra-prediction mode derived using DIMD.

[0256] In the second method, the intra-predicted signal predicted using neural network-based intra-prediction is added as another candidate, in addition to the existing intra-predicted signal predicted using the TIMD-derived intra-prediction mode. The intra-predicted signal predicted using neural network-based intra-prediction is compared with the intra-predicted signal predicted using the TIMD-derived intra-prediction mode during the RDO process in the encoder, and a flag is transmitted in the bitstream to indicate which mode is used. Here, if the intra-predicted signal is predicted using the TIMD-derived intra-prediction mode, the block's intra-prediction mode representation is the TIMD-derived intra-prediction mode; if the intra-predicted signal is predicted using neural network-based intra-prediction, the block's intra-prediction mode representation is either the intra-predicted mode predicted by the neural network or the intra-predicted mode derived using TIMD.

[0257] Intra-frame prediction based on neural networks combined with GPM According to one or more embodiments of this disclosure, neural network-based intra-frame prediction is combined with GPM. Different methods can be used to achieve this goal.

[0258] In the first approach, an intra-predicted signal predicted using a corresponding intra-predicted mode from the original intra-predicted list is replaced with an intra-predicted signal predicted using a neural network-based intra-predicted signal. Currently, in GPM, the intra-predicted list is 3 in length, and the neural network-based intra-predicted signal can replace the first, second, or third intra-predicted signal in the original intra-predicted list. Here, if the ultimately selected intra-predicted signal is a neural network-based intra-predicted signal, the intra-predicted mode representation of the block can be either an intra-predicted mode predicted by the neural network or an intra-predicted mode derived using DIMD; otherwise, the intra-predicted mode representation of the block is the original intra-predicted mode.

[0259] In the second method, the intra-predicted signal predicted using neural network-based intra-prediction is added as another candidate besides the intra-predicted signal predicted using the intra-predicted modes in the original intra-prediction list. Currently, the length of the intra-prediction list in GPM is 3; after adding the neural network-based intra-prediction, the length of the intra-prediction list in GPM is 4. The neural network-based intra-prediction can be inserted at the beginning of the original intra-prediction list, after the first position, after the second position, or at the end position. After comparing the four intra-predicted signals in the intra-prediction list during the RDO process in the encoder, the index of the finally selected intra-predicted signal is transmitted in the bitstream as a signal. Here, if the finally selected intra-prediction is based on neural network-based intra-prediction, the intra-predicted mode representation of the block can be the intra-predicted mode predicted by the neural network or the intra-predicted mode derived using DIMD; otherwise, the intra-predicted mode representation of the block is the original intra-predicted mode.

[0260] Intra-frame prediction based on neural networks combined with SGPM According to one or more embodiments of this disclosure, neural network-based intra-frame prediction is combined with SGPM. Different methods can be used to achieve this goal.

[0261] In the first method, an intra-predicted signal predicted using a corresponding intra-predicted mode from the original intra-predicted list is replaced with an intra-predicted signal predicted using a neural network-based intra-predicted signal. Currently, in SGPM, the length of the intra-predicted list is 3, and the neural network-based intra-predicted signal can replace the first, second, or third intra-predicted signal in the original intra-predicted list. Here, if the ultimately selected intra-predicted signal is a neural network-based intra-predicted signal, the intra-predicted mode representation of the block can be an intra-predicted mode predicted by the neural network or an intra-predicted mode derived using DIMD; otherwise, the intra-predicted mode representation of the block is the original intra-predicted mode.

[0262] In the second method, the intra-predicted signal predicted using neural network-based intra-prediction is added as another candidate besides the intra-predicted signal predicted using the intra-predicted modes in the original intra-prediction list. Currently, the length of the intra-prediction list in SGPM is 3; after adding the neural network-based intra-prediction, the length of the intra-prediction list in SGPM is 4. The neural network-based intra-prediction can be inserted at the beginning of the original intra-prediction list, after the first position, after the second position, or at the end position. After comparing the four intra-predicted signals in the intra-prediction list during the RDO process in the encoder, the index of the finally selected intra-predicted signal is transmitted in the bitstream as a signal. Here, if the finally selected intra-prediction is based on neural network-based intra-prediction, the intra-predicted mode representation of the block can be the intra-predicted mode predicted by the neural network or the intra-predicted mode derived using DIMD; otherwise, the intra-predicted mode representation of the block is the original intra-predicted mode.

[0263] Intra-frame prediction based on neural networks combined with TIMD According to one or more embodiments of this disclosure, neural network-based intra-frame prediction is combined with TIMD. Different methods can be used to achieve this goal.

[0264] In the first method, if the intra-prediction mode derived from TIMD is equal to the intra-prediction mode representation based on neural networks, or if one of the intra-prediction modes in TIMD fusion is equal to the intra-prediction mode representation based on neural networks, then the corresponding intra-prediction mode prediction result is replaced by the intra-prediction result based on neural networks. This corresponding intra-prediction mode prediction result is used as the final result or as part of the fusion process. Otherwise, the prediction result obtained using the original TIMD is saved. The intra-prediction mode representation based on neural networks can be an intra-prediction mode predicted by a neural network or an intra-prediction mode derived using DIMD.

[0265] In the second method, when calculating the SATD between the predicted and reconstructed samples of the template, if an intra-prediction mode during the intra-prediction mode scanning process is equal to the intra-prediction mode representation based on the neural network, then the corresponding predicted sample of the template is generated using the neural network-based intra-prediction. If the final determined intra-prediction mode is the intra-prediction mode representation based on the neural network, or if the intra-prediction mode representation based on the neural network is included in the TIMD fusion process, then the corresponding intra-prediction mode prediction result is generated using the neural network-based intra-prediction. Otherwise, the prediction result obtained using the original TIMD is saved. The intra-prediction mode representation based on the neural network can be an intra-prediction mode predicted by the neural network or an intra-prediction mode derived using DIMD.

[0266] Intra-frame prediction based on neural networks combined with DIMD According to one or more embodiments of this disclosure, intra-frame prediction based on neural networks is combined with DIMD. Different methods can be used to achieve this goal.

[0267] In the first method, if an intra-prediction mode in DIMD fusion equals the intra-prediction mode representation of the neural network-based intra-prediction mode, then the corresponding intra-prediction mode prediction result is replaced by the neural network-based intra-prediction result, which is used as part of the fusion process. Otherwise, the prediction result obtained using the original DIMD is saved. The intra-prediction mode representation of the neural network-based intra-prediction mode can be an intra-prediction mode predicted by the neural network or an intra-prediction mode derived using DIMD.

[0268] Matrix-based position-related intra-frame prediction combined with DIMD According to one or more embodiments of this disclosure, matrix-based position-related intra-frame prediction is combined with DIMD. Different methods can be used to achieve this objective.

[0269] In the first method, if an intra-prediction mode in DIMD fusion is equal to an intra-prediction mode replaced by a matrix-based position-related intra-prediction mode, then the prediction result of the corresponding intra-prediction mode is replaced by the matrix-based position-related intra-prediction result, which is used as part of the fusion process. Otherwise, the prediction result obtained using the original DIMD is saved.

[0270] Figure 24A computing environment 2610 coupled to a user interface 2650 is shown. The computing environment 2610 may be part of a data processing server. The computing environment 2610 includes a processor 2620, memory 2630, and input / output (I / O) interface 2640.

[0271] Processor 2620 typically controls the overall operation of computing environment 2610, such as operations associated with display, data acquisition, data communication, and image processing. Processor 2620 may include one or more processors for executing instructions to perform all or some of the steps in the methods described above. Furthermore, processor 2620 may include one or more modules that facilitate interaction between processor 2620 and other components. The processor may be a central processing unit (CPU), microprocessor, microcontroller, graphics processing unit (GPU), etc.

[0272] Memory 2630 is configured to store various types of data to support the operation of computing environment 2610. Memory 2630 may include predefined software 2632. Examples of such data include instructions for any application or method operating on computing environment 2610, video datasets, image data, etc. Memory 2630 can be implemented using any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0273] I / O interface 2640 provides an interface between processor 2620 and peripheral interface modules (such as keyboard, click wheel, buttons, etc.). Buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. I / O interface 2640 can be coupled to encoders and decoders.

[0274] In an embodiment, a non-transitory computer-readable storage medium is also provided, including, for example, a memory 2630 containing multiple programs and / or storing a bitstream generated by the above-described encoding method or a bitstream to be decoded by the above-described decoding method. The multiple programs can be executed by a processor 2620 in a computing environment 2610 to perform the above-described methods. In an embodiment, the multiple programs can be executed by a processor 2620 in a computing environment 2610 to (e.g., from...) Figure 2The video encoder 20 in the computing environment 2610 receives a bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or one or more associated syntax elements, etc.), and can also be executed by the processor 2620 in the computing environment 2610 to perform the above-described decoding method based on the received bitstream or data stream. In another example, multiple programs can be executed by the processor 2620 in the computing environment 2610 to perform the above-described encoding method to encode video information (e.g., video blocks representing video frames, and / or one or more associated syntax elements, etc.) into a bitstream or data stream, and can also be executed by the processor 2620 in the computing environment 2610 to (e.g., to...) Figure 3 The video decoder 30 in the middle sends the bit stream or data stream. Alternatively, a non-transitory computer-readable storage medium may store data generated by an encoder (e.g., Figure 2 The video encoder 20 in the video encoder (e.g., the one described above) generates the video for the decoder (e.g., the one described above) using the encoding method described above. Figure 3 The video decoder 30 in the video decoder uses a bitstream or data stream that includes encoded video information (e.g., video blocks representing encoded video frames, and / or one or more associated syntax elements, etc.) when decoding video data. Non-transitory computer-readable storage media may be, for example, ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0275] In one embodiment, a bitstream generated by the above-described encoding method or a bitstream to be decoded by the above-described decoding method is provided. In another embodiment, a bitstream comprising encoded video information generated by the above-described encoding method or encoded video information to be decoded by the above-described decoding method is provided.

[0276] In one embodiment, a computing device is also provided, comprising: one or more processors (e.g., processor 2620); and a non-transitory computer-readable storage medium or memory 2630 therein storing a plurality of programs executable by the one or more processors, wherein the one or more processors are configured to perform the methods described above when executing the plurality of programs.

[0277] In one embodiment, a computer program product having instructions for storing or transmitting a bitstream, the bitstream including encoded video information generated by the encoding method described above or encoded video information to be decoded by the decoding method described above, is also provided. In another embodiment, a computer program product including, for example, multiple programs stored in a memory 2630, the multiple programs being executable by a processor 2620 in a computing environment 2610 to perform the methods described above, is also provided. For example, the computer program product may include a non-transitory computer-readable storage medium.

[0278] In an embodiment, the computing environment 2610 may be implemented by one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components for performing the methods described above.

[0279] In one embodiment, a method for storing a bitstream is also provided, comprising: storing the bitstream on a digital storage medium, wherein the bitstream includes encoded video information generated by the above-described encoding method or encoded video information to be decoded by the above-described decoding method.

[0280] In one embodiment, a method for transmitting a bitstream generated by the encoder described above is also provided. In another embodiment, a method for receiving a bitstream to be decoded by the decoder described above is also provided.

[0281] Figure 25 This is a flowchart illustrating a video decoding method 2500 according to some embodiments of the present disclosure. The video decoding method 2500 includes steps S2502-S2506.

[0282] In step S2502, spatial neighboring reference points of the current block are obtained.

[0283] In step S2504, the first intra-frame prediction signal is determined using a neural network-based intra-frame prediction mode based on the spatially nearest reference points of the current block.

[0284] In step S2506, the first intra-frame prediction signal is used as an intra-frame prediction signal or a candidate intra-frame prediction signal for one or more prediction tools.

[0285] According to some embodiments, one or more prediction tools include Inter-Frame and Intra-Frame Joint Prediction (CIIP), and wherein step S2506, using a first intra-frame prediction signal as an intra-frame prediction signal or candidate intra-frame prediction signal for one or more prediction tools, includes: replacing the intra-frame prediction signal determined by intra-frame prediction mode derived using template-based intra-frame mode derivation TIMD in CIIP with the first intra-frame prediction signal, wherein the intra-frame prediction mode of the current block includes one of an intra-frame prediction mode obtained using TIMD, an intra-frame prediction mode determined based on a neural network, or an intra-frame prediction mode obtained by deriving DIMD using decoder-side intra-frame mode derivation.

[0286] According to some embodiments, one or more prediction tools include inter-frame and intra-frame joint prediction (CIIP), and step S2506, using the first intra-frame prediction signal as an intra-frame prediction signal or candidate intra-frame prediction signal for one or more prediction tools, includes: determining a candidate list, the candidate list including intra-frame prediction signals determined by intra-frame prediction modes derived using template-based intra-frame modes (TIMD), and the first intra-frame prediction signal.

[0287] According to some embodiments, method 2500 further includes: receiving a flag indicating a target intra-prediction signal from a bitstream; determining a target intra-prediction signal from a candidate list based on the flag; determining that the intra-prediction mode of the current block is an intra-prediction mode obtained using TIMD in response to the determined target intra-prediction signal being an intra-prediction signal determined using an intra-prediction mode obtained using TIMD; and determining that the intra-prediction mode of the current block is one of an intra-prediction mode based on a neural network or an intra-prediction mode obtained by deriving DIMD using a decoder-side intra-prediction mode inference.

[0288] According to some embodiments, one or more prediction tools include a geometric partitioning mode (GPM), and wherein step S2506, using the first intra-prediction signal as an intra-prediction signal or candidate intra-prediction signal for one or more prediction tools, includes: determining a candidate list, wherein one or more original intra-prediction signals in the candidate list are replaced by the first intra-prediction signal.

[0289] According to some embodiments, one or more prediction tools include a geometric partitioning mode (GPM), and step S2506, using the first intra-prediction signal as an intra-prediction signal or candidate intra-prediction signal for one or more prediction tools, includes: determining a candidate list, wherein the candidate list includes one or more original intra-prediction signals and the first intra-prediction signal to be added, wherein the first intra-prediction signal is added to the head of the candidate list or added after one of the original intra-prediction signals.

[0290] According to some embodiments, method 2500 further includes: receiving from a bitstream a flag indicating a target intra-prediction signal; determining the target intra-prediction signal from a candidate list based on the flag; and, in response to determining that the target intra-prediction signal is the first intra-prediction signal in the candidate list, determining that the intra-prediction mode of the current block is one of an intra-prediction mode based on a neural network or an intra-prediction mode obtained by deriving DIMD using a decoder-side intra-prediction mode.

[0291] According to some embodiments, one or more prediction tools include a Spatial Geometric Partitioning Mode (SGPM), and wherein step S2506, using the first intra-prediction signal as an intra-prediction signal or candidate intra-prediction signal for one or more prediction tools, includes: determining a candidate list, wherein one or more original intra-prediction signals in the candidate list are replaced by the first intra-prediction signal.

[0292] According to some embodiments, one or more prediction tools include a Spatial Geometric Partitioning Mode (SGPM), and step S2506, using the first intra-prediction signal as an intra-prediction signal or candidate intra-prediction signal for one or more prediction tools, includes: determining a candidate list, wherein the candidate list includes one or more original intra-prediction signals and the first intra-prediction signal to be added, wherein the first intra-prediction signal is added to the head of the candidate list or added after one of the original intra-prediction signals.

[0293] According to some embodiments, method 2500 further includes: receiving from a bitstream a flag indicating a target intra-prediction signal; determining the target intra-prediction signal from a candidate list based on the flag; and, in response to determining that the target intra-prediction signal is the first intra-prediction signal in the candidate list, determining that the intra-prediction mode of the current block is one of an intra-prediction mode based on a neural network or an intra-prediction mode obtained by deriving DIMD using a decoder-side intra-prediction mode.

[0294] According to some embodiments, one or more prediction tools include Template-Based Intra-Modal Derivation (TIMD), and step S2506, using a first intra-prediction signal as an intra-prediction signal or candidate intra-prediction signal for one or more prediction tools, includes: replacing the intra-prediction signal determined using the intra-prediction mode obtained using TIMD with the first intra-prediction signal in response to the intra-prediction mode obtained using TIMD being equal to the neural network-based intra-prediction mode; and replacing the intra-prediction signal determined using the one or more intra-prediction modes with the first intra-prediction signal in response to one or more intra-prediction modes being equal to the neural network-based intra-prediction mode during the TIMD fusion process.

[0295] According to some embodiments, one or more prediction tools include template-based intra-frame pattern derivation TIMD, and step S2506, using a first intra-frame prediction signal as an intra-frame prediction signal or candidate intra-frame prediction signal for one or more prediction tools, includes: in response to one or more intra-frame prediction modes in the TIMD fusion process being equal to a neural network-based intra-frame prediction mode, replacing the intra-frame prediction signal determined using the one or more intra-frame prediction modes with the first intra-frame prediction signal to calculate a cost function between the intra-frame prediction signal and the reconstructed signal; and determining the weights of one or more intra-frame prediction modes in the TIMD fusion process based on the cost function.

[0296] According to some embodiments, one or more prediction tools include decoder-side intra-mode derivation DIMD, and wherein step S2506, using a first intra-prediction signal as an intra-prediction signal or candidate intra-prediction signal for one or more prediction tools, includes: in response to one or more intra-prediction modes in the DIMD fusion process being equal to a neural network-based intra-prediction mode, replacing the intra-prediction signal determined using the one or more intra-prediction modes with the first intra-prediction signal.

[0297] Figure 26 This is a flowchart illustrating a video decoding method 2600 according to some embodiments of the present disclosure. The video decoding method 2600 includes steps S2602-S2606.

[0298] In step S2602, a weight matrix is ​​determined for the spatially nearest reference samples of the current block.

[0299] In step S2604, the intra-frame prediction signal is determined based on the weight matrix and spatially nearby reference samples.

[0300] In step S2606, in response to one or more intra-prediction modes in the decoder-side intra-mode derivation DIMD fusion process being equal to the intra-prediction mode for the weight matrix, the intra-prediction signal determined using the one or more intra-prediction modes is replaced with the first intra-prediction signal.

[0301] Figure 27 This is a flowchart illustrating a video encoding method 2700 according to some embodiments of the present disclosure. The video encoding method 2700 includes steps S2702-S2706.

[0302] In step S2702, spatial neighboring reference points of the current block are obtained.

[0303] In some embodiments, the spatially nearest reference point can be the reference point obtained above for intra-frame prediction based on neural networks in NNVC.

[0304] In step S2704, the first intra-frame prediction signal is determined using a neural network-based intra-frame prediction mode based on the spatially nearest reference samples of the current block.

[0305] In some embodiments, the method described above for neural network-based intra-prediction in NNVC can be used to determine the first intra-prediction signal using a neural network-based intra-prediction mode.

[0306] In some embodiments, the intra-prediction signal includes intra-prediction sample values.

[0307] In step S2706, the first intra-frame prediction signal is used as an intra-frame prediction signal or a candidate intra-frame prediction signal for one or more prediction tools.

[0308] According to embodiments of this disclosure, incorporating neural network-based intra-frame prediction modes and prediction signals into video prediction can further improve encoding and decoding performance.

[0309] According to some embodiments, one or more prediction tools include Inter-Frame and Intra-Frame Joint Prediction (CIIP), and wherein step S2706, using a first intra-frame prediction signal as an intra-frame prediction signal or candidate intra-frame prediction signal for one or more prediction tools, includes: replacing the intra-frame prediction signal determined by intra-frame prediction mode derived using template-based intra-frame mode derivation TIMD in CIIP with the first intra-frame prediction signal, wherein the intra-frame prediction mode of the current block includes one of an intra-frame prediction mode obtained using TIMD, an intra-frame prediction mode determined based on a neural network, or an intra-frame prediction mode obtained by deriving DIMD using decoder-side intra-frame mode derivation.

[0310] According to some embodiments, one or more prediction tools include inter-frame and intra-frame joint prediction (CIIP), and wherein step S2706, using the first intra-frame prediction signal as an intra-frame prediction signal or candidate intra-frame prediction signal for one or more prediction tools, includes: adding the first intra-frame prediction signal to a candidate list, the candidate list including intra-frame prediction signals determined by intra-frame prediction modes derived using template-based intra-frame mode derivation (TIMD).

[0311] According to some embodiments, method 2700 further includes: determining a target intra-prediction signal from a candidate list based on rate-distortion optimized RDO parameters; determining that the intra-prediction mode of the current block is an intra-prediction mode obtained using TIMD in response to the determined target intra-prediction signal being an intra-prediction signal determined using an intra-prediction mode obtained using TIMD; determining that the intra-prediction mode of the current block is one of an intra-prediction mode based on a neural network or an intra-prediction mode obtained by deriving DIMD using a decoder-side intra-prediction mode in response to the determined target intra-prediction signal being a first intra-prediction signal; and encoding a flag indicating the target intra-prediction signal into the video bitstream.

[0312] For example, the distortion term in the rate-distortion optimization (RDO) parameters can be determined by the sum of absolute differences (SAD), the sum of squared differences (SSD), or other difference measures.

[0313] According to some embodiments, one or more prediction tools include a geometric partitioning mode (GPM), and step S2706, using a first intra-prediction signal as an intra-prediction signal or candidate intra-prediction signal for one or more prediction tools, includes: replacing an original intra-prediction signal in a candidate list with the first intra-prediction signal, the candidate list including one or more original intra-prediction signals.

[0314] According to some embodiments, one or more prediction tools include a geometric partitioning mode (GPM), and step S2706, using the first intra-prediction signal as an intra-prediction signal or candidate intra-prediction signal for one or more prediction tools, includes: adding the first intra-prediction signal to a candidate list, the candidate list including one or more original intra-prediction signals, wherein the first intra-prediction signal is added to the head of the candidate list or added after one of the original intra-prediction signals.

[0315] According to some embodiments, method 2700 further includes: determining a target intra-prediction signal from a candidate list based on rate-distortion optimized RDO parameters; in response to determining that the target intra-prediction signal is the first intra-prediction signal in the candidate list, determining that the intra-prediction mode of the current block is one of an intra-prediction mode based on a neural network or an intra-prediction mode derived from DIMD using decoder-side intra-prediction mode; and encoding a flag indicating the target intra-prediction signal into the video bitstream.

[0316] According to some embodiments, one or more prediction tools include a Spatial Geometric Partitioning Mode (SGPM), and wherein step S2706, using a first intra-prediction signal as an intra-prediction signal or candidate intra-prediction signal for one or more prediction tools, includes: replacing an original intra-prediction signal in a candidate list with the first intra-prediction signal, the candidate list including one or more original intra-prediction signals.

[0317] According to some embodiments, one or more prediction tools include a Spatial Geometric Partitioning Mode (SGPM), and wherein step S2706, using the first intra-prediction signal as an intra-prediction signal or candidate intra-prediction signal for one or more prediction tools, includes: adding the first intra-prediction signal to a candidate list, the candidate list including one or more original intra-prediction signals, wherein the first intra-prediction signal is added to the head of the candidate list or added after one of the original intra-prediction signals.

[0318] According to some embodiments, method 2700 further includes: determining a target intra-prediction signal from a candidate list based on rate-distortion optimized RDO parameters; in response to determining that the target intra-prediction signal is the first intra-prediction signal in the candidate list, determining that the intra-prediction mode of the current block is one of an intra-prediction mode based on a neural network or an intra-prediction mode derived from DIMD using decoder-side intra-prediction mode; and encoding a flag indicating the target intra-prediction signal into the video bitstream.

[0319] According to some embodiments, one or more prediction tools include Template-Based Intra-Modal Derivation (TIMD), and step S2706, using a first intra-prediction signal as an intra-prediction signal or candidate intra-prediction signal for one or more prediction tools, includes: replacing the intra-prediction signal determined using the intra-prediction mode obtained using TIMD with the first intra-prediction signal in response to the intra-prediction mode obtained using TIMD being equal to the neural network-based intra-prediction mode; and replacing the intra-prediction signal determined using the one or more intra-prediction modes with the first intra-prediction signal in response to one or more intra-prediction modes being equal to the neural network-based intra-prediction mode during the TIMD fusion process.

[0320] According to some embodiments, one or more prediction tools include template-based intra-frame pattern derivation TIMD, and wherein step S2706, using a first intra-frame prediction signal as an intra-frame prediction signal or candidate intra-frame prediction signal for one or more prediction tools, includes: in response to one or more intra-frame prediction modes in the TIMD fusion process being equal to a neural network-based intra-frame prediction mode, replacing the intra-frame prediction signal determined using the one or more intra-frame prediction modes with the first intra-frame prediction signal to calculate a cost function between the intra-frame prediction signal and the reconstructed signal; and determining the weights of one or more intra-frame prediction modes in the TIMD fusion process based on the cost function.

[0321] According to some embodiments, one or more prediction tools include decoder-side intra-mode derivation (DIMD), and step S2706, using a first intra-prediction signal as an intra-prediction signal or candidate intra-prediction signal for one or more prediction tools, includes: in response to one or more intra-prediction modes in the DIMD fusion process being equal to a neural network-based intra-prediction mode, replacing the intra-prediction signal determined using the one or more intra-prediction modes with the first intra-prediction signal.

[0322] Figure 28 This is a flowchart illustrating a video encoding method 2800 according to some embodiments of the present disclosure. The video encoding method 2800 includes steps S2802-S2806.

[0323] In step S2802, a weight matrix is ​​determined for the spatially nearest reference samples of the current block.

[0324] In step S2804, the intra-frame prediction signal is determined based on the weight matrix and spatially nearby reference samples.

[0325] In step S2806, in response to one or more intra-prediction modes in the decoder-side intra-mode derivation DIMD fusion process being equal to the intra-prediction mode for the weight matrix, the intra-prediction signal determined using the one or more intra-prediction modes is replaced with the first intra-prediction signal.

[0326] According to embodiments of this disclosure, importing matrix-based position-related intra-frame prediction and prediction signals into video prediction can further improve encoding and decoding performance.

[0327] In some embodiments, the matrix-based position-related intra-frame prediction described above can be used to replace the method described in the existing conventional intra-frame mode (PDP) to determine the weight matrix, and the intra-frame prediction signal including the prediction sample can be determined based on the weight matrix and spatially nearby reference samples.

[0328] According to some embodiments, a computing device is provided, including: one or more processors; and a memory coupled to the one or more processors, wherein the memory is configured to store instructions executable by the one or more processors, which, when executing the instructions, cause the computing device to perform the methods described above.

[0329] According to some embodiments, a non-transitory computer-readable storage medium is provided, storing a bit stream generated by instructions that, when executed by a computing device having one or more processors, cause one or more processors to perform the above-described method for video encoding.

[0330] According to some embodiments, a method for storing a bitstream is provided, comprising: generating a bitstream according to the method described above for video encoding; and storing the bitstream.

[0331] According to some embodiments, a computer program product is provided, including instructions that, when executed by one or more processors of a computing device, cause the computing device to perform the methods described above.

[0332] The description in this disclosure is presented for illustrative purposes and is not intended to be exhaustive or limited to this disclosure. Many modifications, variations, and alternative embodiments will be apparent to those skilled in the art from the teachings presented in the foregoing description and the associated drawings.

[0333] Unless otherwise specified, the order of steps in the method according to this disclosure is intended to be illustrative only, and the steps of the method according to this disclosure are not limited to the specific order described above, but may be changed according to actual circumstances. Furthermore, at least one step in the method according to this disclosure may be adjusted, combined, or omitted as needed.

[0334] The examples were chosen and described to explain the principles of this disclosure and to enable others skilled in the art to understand the various embodiments of this disclosure, and preferably to utilize the basic principles and various embodiments with various modifications suitable for the intended particular purpose. Therefore, it should be understood that the scope of this disclosure is not limited to the specific examples of the disclosed embodiments, and that modifications and other embodiments are intended to be included within the scope of this disclosure.

Claims

1. A method for video decoding, comprising: Obtain the spatial neighboring reference points of the current block; Based on the spatially nearest reference samples of the current block, the first intra-frame prediction signal is determined using a neural network-based intra-frame prediction mode. as well as The first intra-frame prediction signal is used as an intra-frame prediction signal or a candidate intra-frame prediction signal for one or more prediction tools.

2. The method according to claim 1, wherein, The one or more prediction tools include inter-frame and intra-frame joint prediction (CIIP), and wherein using the first intra-frame prediction signal as an intra-frame prediction signal or a candidate intra-frame prediction signal for the one or more prediction tools includes: The intra-prediction signal determined by the intra-prediction mode derived from the template-based intra-mode derivation TIMD in the CIIP is replaced with the first intra-prediction signal. The intra-prediction mode of the current block includes one of the following: the intra-prediction mode obtained using the TIMD, the intra-prediction mode determined based on the neural network, or the intra-prediction mode obtained by deriving DIMD using the decoder-side intra-prediction mode.

3. The method according to claim 1, wherein, The one or more prediction tools include Inter-Frame and Intra-Frame Joint Prediction (CIIP), and wherein using the first intra-frame prediction signal as an intra-frame prediction signal or a candidate intra-frame prediction signal for the one or more prediction tools includes: A candidate list is determined, which includes the intra-prediction signal determined by the intra-prediction mode obtained by deriving TIMD using a template-based intra-prediction mode, and the first intra-prediction signal.

4. The method according to claim 3, further comprising: Receive a flag from the bitstream that indicates the intra-frame prediction signal of the target frame; Based on the flag, the target intra-frame prediction signal is determined from the candidate list; In response to the determination that the target intra-prediction signal is an intra-prediction signal determined using the intra-prediction mode obtained by the TIMD, the intra-prediction mode of the current block is determined to be the intra-prediction mode obtained by the TIMD. as well as In response to the determined target intra-prediction signal being the first intra-prediction signal, the intra-prediction mode of the current block is determined to be either the intra-prediction mode based on the neural network or the intra-prediction mode obtained by deriving DIMD using the decoder-side intra-prediction mode.

5. The method according to claim 1, wherein, The one or more prediction tools include a geometric partitioning mode (GPM), and wherein using the first intra-prediction signal as an intra-prediction signal or a candidate intra-prediction signal for the one or more prediction tools includes: A candidate list is determined, wherein one or more original intra-prediction signals in the candidate list are replaced by the first intra-prediction signal.

6. The method according to claim 1, wherein, The one or more prediction tools include a geometric partitioning mode (GPM), and wherein using the first intra-prediction signal as an intra-prediction signal or a candidate intra-prediction signal for the one or more prediction tools includes: A candidate list is determined, wherein the candidate list includes one or more original intra-prediction signals and the first intra-prediction signal to be added. The first intra-frame prediction signal is either added to the head of the candidate list or added after one of the one or more original intra-frame prediction signals.

7. The method according to any one of claims 5-6, further comprising: Receive a flag from the bitstream that indicates the intra-frame prediction signal of the target frame; Based on the flag, the target intra-frame prediction signal is determined from the candidate list; In response to determining that the target intra-prediction signal is the first intra-prediction signal in the candidate list, the intra-prediction mode of the current block is determined to be either an intra-prediction mode based on the neural network or an intra-prediction mode obtained by deriving DIMD using the decoder-side intra-prediction mode.

8. The method according to claim 1, wherein, The one or more prediction tools include a Spatial Geometric Partitioning Mode (SGPM), and wherein using the first intra-prediction signal as an intra-prediction signal or candidate intra-prediction signal for the one or more prediction tools includes: A candidate list is determined, wherein one or more original intra-prediction signals in the candidate list are replaced by the first intra-prediction signal.

9. The method according to claim 1, wherein, The one or more prediction tools include a Spatial Geometric Partitioning Mode (SGPM), and wherein using the first intra-prediction signal as an intra-prediction signal or candidate intra-prediction signal for the one or more prediction tools includes: A candidate list is determined, wherein the candidate list includes one or more original intra-prediction signals and the first intra-prediction signal to be added. The first intra-frame prediction signal is either added to the head of the candidate list or added after one of the one or more original intra-frame prediction signals.

10. The method according to any one of claims 8-9, further comprising: Receive a flag from the bitstream that indicates the intra-frame prediction signal of the target frame; Based on the flag, the target intra-frame prediction signal is determined from the candidate list; In response to determining that the target intra-prediction signal is the first intra-prediction signal in the candidate list, the intra-prediction mode of the current block is determined to be either an intra-prediction mode based on the neural network or an intra-prediction mode obtained by deriving DIMD using the decoder-side intra-prediction mode.

11. The method according to claim 1, wherein, The one or more prediction tools include Template-Based Intra-Frame Pattern Derivation (TIMD), and wherein using the first intra-frame prediction signal as an intra-frame prediction signal or candidate intra-frame prediction signal for the one or more prediction tools includes: In response to the intra-prediction mode obtained using the TIMD being equal to the neural network-based intra-prediction mode, the intra-prediction signal determined using the intra-prediction mode obtained using the TIMD is replaced with the first intra-prediction signal; and In response to one or more intra-prediction modes in the TIMD fusion process being equal to a neural network-based intra-prediction mode, the intra-prediction signal determined using the one or more intra-prediction modes is replaced with the first intra-prediction signal.

12. The method according to claim 1, wherein, The one or more prediction tools include Template-Based Intra-Frame Pattern Derivation (TIMD), and wherein using the first intra-frame prediction signal as an intra-frame prediction signal or candidate intra-frame prediction signal for the one or more prediction tools includes: In response to one or more intra-prediction modes in the TIMD fusion process being equal to a neural network-based intra-prediction mode, the intra-prediction signal determined using the one or more intra-prediction modes is replaced with the first intra-prediction signal to calculate a cost function between the intra-prediction signal and the reconstructed signal; and Based on the cost function, the weights of one or more intra-frame prediction modes in the TIMD fusion process are determined.

13. The method according to claim 1, wherein, The one or more prediction tools include decoder-side intra-frame mode derivation (DIMD), and wherein using the first intra-frame prediction signal as an intra-frame prediction signal or candidate intra-frame prediction signal for the one or more prediction tools includes: In response to one or more intra-prediction modes in the DIMD fusion process being equal to a neural network-based intra-prediction mode, the intra-prediction signal determined using the one or more intra-prediction modes is replaced with the first intra-prediction signal.

14. A method for video decoding, comprising: Determine the weight matrix for spatially adjacent reference samples of the current block; The intra-frame prediction signal is determined based on the weight matrix and the spatially nearest reference samples. as well as In response to one or more intra-prediction modes being equal to the intra-prediction mode for the weight matrix during the decoder-side intra-mode derivation DIMD fusion process, the intra-prediction signal determined using the one or more intra-prediction modes is replaced with the first intra-prediction signal.

15. A method for video encoding, comprising: Obtain the spatial neighboring reference points of the current block; Based on the spatially nearest reference samples of the current block, the first intra-frame prediction signal is determined using a neural network-based intra-frame prediction mode. as well as The first intra-frame prediction signal is used as an intra-frame prediction signal or a candidate intra-frame prediction signal for one or more prediction tools.

16. The method according to claim 15, wherein, The one or more prediction tools include inter-frame and intra-frame joint prediction (CIIP), and wherein using the first intra-frame prediction signal as an intra-frame prediction signal or a candidate intra-frame prediction signal for the one or more prediction tools includes: The intra-prediction signal determined by the intra-prediction mode derived from the template-based intra-mode derivation TIMD in the CIIP is replaced with the first intra-prediction signal. The intra-prediction mode of the current block includes one of the following: the intra-prediction mode obtained using the TIMD, the intra-prediction mode determined based on the neural network, or the intra-prediction mode obtained by deriving DIMD using the decoder-side intra-prediction mode.

17. The method according to claim 15, wherein, The one or more prediction tools include Inter-Frame and Intra-Frame Joint Prediction (CIIP), and wherein using the first intra-frame prediction signal as an intra-frame prediction signal or a candidate intra-frame prediction signal for the one or more prediction tools includes: The first intra-prediction signal is added to a candidate list, which includes intra-prediction signals determined by intra-prediction modes derived using template-based intra-mode derivation of TIMD.

18. The method of claim 17, further comprising: Based on rate-distortion optimization of RDO parameters, the target intra-frame prediction signal is determined from the candidate list; In response to the determination that the target intra-prediction signal is an intra-prediction signal determined using the intra-prediction mode obtained by the TIMD, the intra-prediction mode of the current block is determined to be the intra-prediction mode obtained by the TIMD. In response to the determination that the target intra-prediction signal is the first intra-prediction signal, the intra-prediction mode of the current block is determined to be either the intra-prediction mode based on the neural network or the intra-prediction mode obtained by deriving DIMD using the decoder-side intra-prediction mode. as well as The flag indicating the target intra-frame prediction signal is encoded into the video bitstream.

19. The method according to claim 15, wherein, The one or more prediction tools include a geometric partitioning mode (GPM), and wherein using the first intra-prediction signal as an intra-prediction signal or a candidate intra-prediction signal for the one or more prediction tools includes: The original intra-prediction signal in the candidate list is replaced with the first intra-prediction signal, the candidate list including one or more original intra-prediction signals.

20. The method of claim 15, wherein, The one or more prediction tools include a geometric partitioning mode (GPM), and wherein using the first intra-prediction signal as an intra-prediction signal or a candidate intra-prediction signal for the one or more prediction tools includes: The first intra-frame prediction signal is added to the candidate list, which includes one or more original intra-frame prediction signals. The first intra-frame prediction signal is either added to the head of the candidate list or added after one of the one or more original intra-frame prediction signals.

21. The method according to any one of claims 19-20, further comprising: Based on rate-distortion optimization of RDO parameters, the target intra-frame prediction signal is determined from the candidate list; In response to determining that the target intra-prediction signal is the first intra-prediction signal in the candidate list, the intra-prediction mode of the current block is determined to be one of the intra-prediction mode based on the neural network or the intra-prediction mode obtained by deriving DIMD using the decoder-side intra-prediction mode. as well as The flag indicating the target intra-frame prediction signal is encoded into the video bitstream.

22. The method according to claim 15, wherein, The one or more prediction tools include a Spatial Geometric Partitioning Mode (SGPM), and wherein using the first intra-prediction signal as an intra-prediction signal or candidate intra-prediction signal for the one or more prediction tools includes: The original intra-prediction signal in the candidate list is replaced with the first intra-prediction signal, the candidate list including one or more original intra-prediction signals.

23. The method according to claim 15, wherein, The one or more prediction tools include a Spatial Geometric Partitioning Mode (SGPM), and wherein using the first intra-prediction signal as an intra-prediction signal or candidate intra-prediction signal for the one or more prediction tools includes: The first intra-frame prediction signal is added to the candidate list, which includes one or more original intra-frame prediction signals. The first intra-frame prediction signal is either added to the head of the candidate list or added after one of the one or more original intra-frame prediction signals.

24. The method according to any one of claims 22-23, further comprising: Based on rate-distortion optimization of RDO parameters, the target intra-frame prediction signal is determined from the candidate list; In response to determining that the target intra-prediction signal is the first intra-prediction signal in the candidate list, the intra-prediction mode of the current block is determined to be one of the intra-prediction mode based on the neural network or the intra-prediction mode obtained by deriving DIMD using the decoder-side intra-prediction mode. as well as The flag indicating the target intra-frame prediction signal is encoded into the video bitstream.

25. The method of claim 15, wherein, The one or more prediction tools include Template-Based Intra-Frame Pattern Derivation (TIMD), and wherein using the first intra-frame prediction signal as an intra-frame prediction signal or candidate intra-frame prediction signal for the one or more prediction tools includes: In response to the intra-prediction mode obtained using the TIMD being equal to the neural network-based intra-prediction mode, the intra-prediction signal determined using the intra-prediction mode obtained using the TIMD is replaced with the first intra-prediction signal; and In response to one or more intra-prediction modes in the TIMD fusion process being equal to a neural network-based intra-prediction mode, the intra-prediction signal determined using the one or more intra-prediction modes is replaced with the first intra-prediction signal.

26. The method of claim 15, wherein, The one or more prediction tools include Template-Based Intra-Frame Pattern Derivation (TIMD), and wherein using the first intra-frame prediction signal as an intra-frame prediction signal or candidate intra-frame prediction signal for the one or more prediction tools includes: In response to one or more intra-prediction modes in the TIMD fusion process being equal to a neural network-based intra-prediction mode, the intra-prediction signal determined using the one or more intra-prediction modes is replaced with the first intra-prediction signal to calculate a cost function between the intra-prediction signal and the reconstructed signal; and Based on the cost function, the weights of one or more intra-frame prediction modes in the TIMD fusion process are determined.

27. The method according to claim 15, wherein, The one or more prediction tools include decoder-side intra-frame mode derivation (DIMD), and wherein using the first intra-frame prediction signal as an intra-frame prediction signal or candidate intra-frame prediction signal for the one or more prediction tools includes: In response to one or more intra-prediction modes in the DIMD fusion process being equal to a neural network-based intra-prediction mode, the intra-prediction signal determined using the one or more intra-prediction modes is replaced with the first intra-prediction signal.

28. A method for video encoding, comprising: Determine the weight matrix for spatially adjacent reference samples of the current block; The intra-frame prediction signal is determined based on the weight matrix and the spatially nearest reference samples. as well as In response to one or more intra-prediction modes being equal to the intra-prediction mode for the weight matrix during the decoder-side intra-mode derivation DIMD fusion process, the intra-prediction signal determined using the one or more intra-prediction modes is replaced with the first intra-prediction signal.

29. A computing device, comprising: One or more processors; as well as Memory coupled to the one or more processors, The memory is configured to store instructions executable by the one or more processors, which, when executing the instructions, cause the computing device to perform the method as described in any one of claims 1-28.

30. A non-transitory computer-readable storage medium storing a bit stream generated by instructions, which, when executed by a computing device having one or more processors, cause the one or more processors to perform the method as described in any one of claims 15-28.

31. A method for storing a bit stream, comprising: A bitstream is generated according to the method described in any one of claims 15-28; as well as Store the bit stream.

32. A computer program product comprising instructions that, when executed by one or more processors of a computing device, cause the computing device to perform the method as described in any one of claims 1-28.

Images