Interaction between transform partitioning and primary / secondary transform type selection

Abstract

To provide methods and apparatuses for encoding / decoding video data according to advanced video coding technologies.SOLUTION: A method includes: extracting, from a received coded video bitstream, a transform partition type associated with a data block; and, in response to the transform partition type belonging to a subset of a predefined set of transform partition types each specifying a split pattern for dividing the data block into transform blocks: extracting, as signaled in the coded video bitstream, a transform type of a transform associated with a transform block split from the data block; and performing an inverse transform on the transform block according to the transform type. The transform type belongs to a first predefined set of transform types.SELECTED DRAWING: Figure 17

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 175,897, filed April 16, 2021, and U.S. Nonprovisional Patent Application No. 17 / 568,275, filed January 4, 2022, both of which are incorporated by reference in their entireties herein.

[0002] This disclosure describes a set of advanced video coding techniques. More specifically, the disclosure The proposed technique is a transform partitioning scheme and a primary Includes interaction between / secondary transform type selection. [Background technology]

[0003] The discussion of the background art provided herein is intended to generally present the context of the present disclosure. The inventors' research, insofar as that research is described in this Background section, and other explanations that may not be admitted as prior art at the time of filing of this application. This and other aspects are not admitted explicitly or implicitly as prior art to the present disclosure.

[0004] Video coding and decoding is based on inter-picture prediction with motion compensation. This can be done using a number of different techniques. Uncompressed digital video contains a series of pictures. Each picture can contain, for example, 1920x1080 luma samples and associated full sampling The sequence of pictures has a spatial dimension of 128 or subsampled chrominance samples. For example, fixed or variable picture rates (or frames) of 60 pictures per second or 60 frames per second. Uncompressed video can have a bit rate (also called frame rate). For example, a 1920x1080 pixel resolution, 60 frames per second, Frame rate of 1000 frames per second and 4:2:0 chroma at 8 bits per pixel per color channel. A video with subsampling requires a bandwidth approaching 1.5 Gbit / s. Such a video would require over 600GByte of storage space.

[0005] One purpose of video coding and video decoding is to compress an uncompressed input Compression can be the reduction of redundancy in a video signal. , which can help reduce image quality by more than two orders of magnitude in some cases. Lossless compression means that an exact copy of the original signal is reproduced. This refers to a technique that allows a signal to be reconstructed from the original signal that was compressed by the process. This means that the original video information is not fully preserved during coding and cannot be fully recovered during decoding. When lossy compression is used, the reconstructed The reconstructed signal may not be identical to the original signal, but the distortion between the original and the reconstructed signal may be small. This means that even with some loss of information, the reconstructed signal is sufficient to be useful for the intended purpose. In the case of video, lossy compression is widely used in many applications. The amount of distortion that can be tolerated depends on the application. For example, certain consumer video streaming applications Users of commercial applications may tolerate higher distortion than users of movie and television broadcast applications. The compression ratios achievable by the filtering algorithms are chosen to reflect different distortion tolerances. That is, generally, the higher the distortion tolerance, the higher the loss and the higher the Coding algorithms that result in compression ratios are possible.

[0006] The video encoder and video decoder perform various operations, such as motion compensation, Fourier transform, quantization, Techniques from several broad categories and steps, including image processing, image retrieval, and entropy coding. You can use the technique.

[0007] Video codec technology can include a technique known as intracoding. In intra-coding, sample values ​​are derived from previously reconstructed reference pictures. It is represented without reference to samples or other data. Some video codecs use The image is spatially subdivided into blocks of samples. Every block of samples is an intra-sampled image. If the picture is coded in the Intra pictures and their derivatives such as independent decoder refresh pictures The decoder can be used to reset the decoder state and therefore as the first picture in the encoded video bitstream and in the video session, can be used as a still image. Then, the samples of the block after intra prediction can be transformed to the frequency domain and the transform coefficients so generated can be used as the entropy Before coding, the image can be quantized. Intra prediction is the process of sampling in the pre-transform domain. In some cases, the smaller the DC value after conversion, the greater the AC coefficient. The smaller the number, the smaller the number of quantization steps required to represent the block after entropy coding. This reduces the number of bits required in the download size.

[0008] For example, conventional intracoders such as those known from MPEG-2 generation coding techniques. However, some newer video compression techniques do not use intra prediction. The technique may, for example, be used to detect intra-coded, or spatially adjacent, coding and / or decoding errors. The surrounding samples that precede in decoding order the block of data being decoded or intra-decoded. coding / decoding of blocks based on the packet data and / or metadata Such techniques are hereafter referred to as "intra-prediction" techniques. In some cases, intra prediction relies on references only from the current picture being reconstructed. Note that the reference data used is the same as the original reference picture, and no reference data from other reference pictures is used. .

[0009] Intra prediction can take many different forms. If available for a given video coding technique, the technique used is One or more intra prediction modes can be used for a particular codec. In certain cases, a mode may have submodes and / or or various parameters, such as mode / submode information and video The intra-coding parameters of the blocks can be coded individually or collectively. For a given mode, submode, and / or The codeword to be used for the parameter combination is determined by the intra-prediction codeword. This can affect the efficiency of the bitstream. The entropy coding technique used to convert the image into a binary may also have an impact. do.

[0010] A specific mode of intra prediction was introduced in H.264 and improved in H.265, the joint search model. (JEM), versatile video coding (VVC), and benchmark set (BMS) Newer coding techniques have further improved this. In general, intra prediction uses The predictor block can be formed using adjacent sample values ​​that have been The available values ​​of a particular set of adjacent samples along a certain direction and / or line are used to determine the predictor block. A reference to the direction used is coded in the bitstream. It may be possible to predict the time course of a given event, or it may itself be predicted.

[0011] Referring to FIG. 1A, the bottom right shows the 35 intra-motion video formats specified in H.265. The nine predictor directions specified in H.265 correspond to the 33 possible predictor directions in H.265 (corresponding to the 33 angle modes in the H.265 The arrows converge at point (101) which is the sample being predicted. The arrows represent the neighboring samples that are used to predict the 101 samples from them. For example, the arrow (102) indicates that the sample (101) is adjacent to one or more neighboring samples. The arrows (10 3) is a case where the sample (101) is shifted from one or more adjacent samples to the lower left of the sample (101); This indicates that it is predicted at an angle of 22.5 degrees from the horizontal.

[0012] Still referring to FIG. 1A, at the top left is a positive 4×4 sample (indicated by the thick dashed line). A square block (104) is depicted. The square block (104) contains 16 samples. , respectively, “S”, its position in the Y dimension (e.g., row index), and its position in the X dimension ( For example, sample S21 is labeled by the Y-dimension ( It is the second sample in the x dimension (from the top) and the first sample in the x dimension (from the left). Sample S44 is the fourth sample in block (104) in both the Y and X dimensions. The size of the block is 4x4 samples, hence S44 at the bottom right. An example of a reference sample to follow is further shown. The reference sample is R, for block (104). Each pixel is labeled with its Y position (i.e. row number) and X position (column number). In both H.260 and H.265, predictive samples that neighbor the block being reconstructed are used.

[0013] The intra-picture prediction of block 104 predicts adjacent sub-pictures according to the signaled prediction direction. One may start by copying the reference sample values ​​from a coded sample. The resulting video bitstream indicates the prediction direction of the arrow (102) for this block 104. Signaling is included, i.e., the sample is a signal from one or more predicted samples to the top right, Assume that the projections are at an angle of 45 degrees from the horizontal. In such a case, samples S41, S32, and S2 3, S14 is predicted from the same reference sample R05. Then sample S44 is predicted from the reference sample Predicted from R08.

[0014] In certain cases, in order to calculate the reference sample, the orientation must be evenly divided by 45 degrees. When not cut off, the values ​​of multiple reference samples may be combined, for example by interpolation. .

[0015] The number of possible directions has increased as video coding technology continues to develop. In .264 (2003), for example, nine different directions are available for intra prediction. increased to 33 in H.265 (2013), and JEM / VVC / BMS is up to 65 at the time of this disclosure. It can support the most suitable intra prediction direction. Experimental studies have been carried out to determine the directionality of the image using a specific technique of entropy coding. accept a certain bit penalty for all, and their most appropriate direction is coded with a small number of bits. Furthermore, the direction itself can be used in intra prediction of the decoded neighboring blocks. In some cases, this can be predicted from the neighboring directions.

[0016] Figure 1B shows the increasing number of prediction directions in various coding techniques that have evolved over time. To illustrate, we show a schematic diagram (180) showing the 65 intra prediction directions according to JEM.

[0017] A bit representing an intra-prediction direction in a coded video bitstream. The mapping to the prediction direction may vary depending on the video coding technology. For example, a simple direct mapping of prediction directions to intra prediction modes yields the codeword, the most likely These may range from complex adaptive schemes including highly adaptive modes and similar techniques. In all cases, it is statistically more likely to occur in video content than in any other particular orientation. There may be a specific direction of low intro prediction. Since the goal of video compression is to reduce redundancy, Therefore, in a well-designed video coding technique, the less likely The direction is represented by more bits than the more likely direction.

[0018] Inter-picture prediction, or inter-prediction, can be based on motion compensation. Compensation involves the use of sample data from a previously reconstructed picture or part of it (a reference picture). The data is then spatially shifted in the direction indicated by the motion vector (hereafter MV), It may be used to predict a newly reconstructed picture or picture portion (e.g., a block). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV is , it may have two dimensions X and Y, or three dimensions, the third dimension being (similar to the time dimension) An indication of the reference picture to be used (similar to a reference picture).

[0019] Some video compression techniques use the current MV from another MV, e.g., one that is spatially adjacent to the area being reconstructed and precedes the current MV in decoding order. Then, predictions can be made from other MVs that relate to other areas of the sample data. By using a MV coding scheme, we rely on removing redundancy in correlated MVs. This can significantly reduce the overall amount of data required to MV prediction can work effectively for example in the case of When coding the input video signal derived from a camera (which is Areas larger than the applicable area move in a similar direction in the video sequence. There is a statistical likelihood that the MVs in the neighboring areas will be As a result, the desired motion vector can be predicted using the same motion vector. The actual MV of a given area will be similar or identical to the predicted MV from the surrounding MVs. Furthermore, after entropy coding, the MV is predicted from (one or more) neighboring MVs. is less than the number of bits that would be used if the In some cases, the MV prediction can be performed using the original signal (i.e., the number of samples This may be an example of lossless compression of a signal (i.e., MV) derived from a video stream. In other cases, e.g. due to rounding errors when computing the predictor from several surrounding MVs Additionally, the MV prediction itself may be lossy.

[0020] H.265 / HEVC (ITU-T Rec.H.265, "High Efficiency Video Coding", December 2016 Among the many MV prediction mechanisms specified by H.265, the following are Described below is a technique hereafter referred to as "spatial merging."

[0021] Specifically, referring to FIG. 2, a current block (201) is an encoded block during the motion search process. The coder assumes that the block is predictable from the previous block of the same size but spatially shifted. Instead of coding the MV directly, we divide the MV into A0, A1, and and one of the five surrounding samples, denoted B0, B1, and B2 (202 to 206, respectively). Using the linked MV, one or more reference pictures and associated metadata For example, it can be derived from the last reference picture (in decoding order). MV prediction uses predictors from the same reference picture as the neighboring blocks use. It is possible. Summary of the Invention [Means for solving the problem]

[0022] The present disclosure relates to methods, apparatus, and computers for video encoding and / or video decoding. Various embodiments of a readable storage medium are described.

[0023] According to one aspect, an embodiment of the present disclosure includes a decoder for encoding / decoding video data. The present invention provides a method for decoding coded video bits of a data block, the method comprising: receiving a coded video bitstream; and from the coded video bitstream, Extracting a transformation partition type associated with the data block; The partition type is the division pattern for dividing data blocks into conversion blocks. Each of these transformations belongs to a subset of a predefined set of partition types. In response to the determination, a signal is signaled in the coded video bitstream. The transformation type of the transformation associated with the transformation block split out from the data block, wherein the conversion type is selected from a first predefined set of conversion types. Steps that perform inverse transformation on transformation blocks according to the step and transformation type. This includes the step.

[0024] According to another aspect, an embodiment of the present disclosure provides a method for encoding / decoding video data. The method includes receiving a coded video bitstream of data blocks. and receiving the coded video bitstream and decoding the video data from the coded video bitstream. extracting a transformation partition type associated with the data block; Transformation partitions that belong to a subset of a predefined set of partition types. In response to the type, the coded video bitstream is Extracting a transformation type associated with the transformation block; In response to a conversion partition type that does not belong to a predefined set of and identifying by default the conversion type of the tab block.

[0025] According to another aspect, an embodiment of the present disclosure provides a method for encoding / decoding video data. The method includes receiving a coded video bitstream of data blocks. receiving from the coded video bitstream a block of data; Extracting a transformation type of a transformation associated with the transformation block; A coded video bitstream is generated in response to a transformation type belonging to a predefined set. A stream that extracts the conversion partition type associated with a data block. This includes the step.

[0026] According to another aspect, an embodiment of the present disclosure provides a method for video encoding and / or video decoding, comprising: The present invention provides an apparatus for performing a programmable logic operation on a computer, the apparatus comprising: a memory for storing instructions; and a processor in communication with the memory. When the processor executes the instructions, the processor may cause the device to perform video decoding and / or video The device is configured to perform the above method for video encoding.

[0027] According to yet another aspect, an embodiment of the present disclosure provides a method for decoding and / or encoding a video signal. for video decoding and / or video encoding when executed by a computer A non-transitory computer readable medium storing instructions for causing a computer to carry out the above method. to provide.

[0028] These and other aspects and their implementations are set forth in the drawings, specification, and claims. This will be explained in more detail below.

[0029] Further features, characteristics and properties of the disclosed subject matter are set forth in the following detailed description and the accompanying drawings. and various effects become more apparent. [Brief description of the drawings]

[0030] [Figure 1A] FIG. 13 is a schematic diagram of an example subset of intra-prediction directional modes. [Figure 1B] FIG. 2 illustrates an exemplary intra-prediction direction. [Diagram 2] FIG. 2 is a schematic diagram illustrating a current block and its surrounding spatial merge candidates for motion vector prediction in one example. [Diagram 3] FIG. 3 is a schematic diagram illustrating a simplified block diagram of a communication system (300) according to an example embodiment. [Figure 4] FIG. 4 is a schematic diagram illustrating a simplified block diagram of a communication system (400) according to an example embodiment. [Diagram 5] FIG. 2 is a schematic diagram illustrating a simplified block diagram of a video decoder according to an example embodiment. [Figure 6] FIG. 1 is a schematic diagram illustrating a simplified block diagram of a video encoder according to an example embodiment. [Figure 7] FIG. 2 is a block diagram illustrating a video encoder according to another example embodiment. [Figure 8] FIG. 2 is a block diagram illustrating a video decoder according to another example embodiment. [Figure 9] FIG. 2 illustrates directional intra-prediction modes according to an exemplary embodiment of the present disclosure. [Figure 10] FIG. 2 illustrates a non-directional intra-prediction mode according to an exemplary embodiment of the present disclosure. [Figure 11] 1 illustrates a recursive intra-prediction mode according to an exemplary embodiment of the present disclosure. [Figure 12] 4 illustrates transform block partitioning and scanning of intra-predicted blocks according to an exemplary embodiment of the present disclosure. [Figure 13] 1 illustrates transform block partitioning and scanning of inter-prediction blocks according to an exemplary embodiment of the present disclosure. [Figure 14] 4 illustrates a low frequency non-separable transformation process according to an exemplary embodiment of the present disclosure. [Figure 15] FIG. 2 illustrates various baseline-based intra-prediction schemes, according to an exemplary embodiment of the present disclosure. [Figure 16] FIG. 2 illustrates a non-recursive block division scheme according to an exemplary embodiment of the present disclosure. [Figure 17] 1 shows a flow chart according to an embodiment of the present disclosure. [Figure 18] FIG. 1 is a schematic diagram illustrating a computer system according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] FIG. 3 illustrates a simplified block diagram of a communication system (300) according to one embodiment of the present disclosure. The communication systems (300) communicate with each other via, for example, a network (350). For example, the communication system (300) includes a network (350). In the example of FIG. 3, the system includes a first pair of terminal devices (310) and (320) interconnected via a A first pair of terminal devices 310 and 320 may perform unidirectional transmission of data. For example, The terminal device (310) transmits the packet to the other terminal device (320) via the network (350). (e.g., of a stream of video pictures captured by the terminal device (310)) The encoded video data may include one or more codes. The video data can be transmitted in the form of a coded video bitstream. receives coded video data from the network (350); The decoded video data is decoded to restore the video picture, and the video picture is reproduced according to the restored video data. One-way data transmission is implemented in media serving applications, etc. It can be carried out.

[0032] In another example, the communication system (300) may include a call center, such as may be implemented during video conferencing applications. A second pair of terminal devices (330) and (340) for performing bidirectional transmission of the coded video data. 0). For bidirectional data transmission, in one example, each of the terminal devices (330) and (340) The terminal device communicates with the other terminal device of the terminal devices (330) and (340) via the network (350). for transmitting to the terminal device (e.g., for storing video pictures captured by the terminal device) Each of the terminal devices 330 and 340 may code video data (streams). The terminal also receives the coding transmitted by the other of the terminals (330) and (340). The video data is received, and the coded video data is decoded to generate video pictures. and displaying the video picture on an accessible display device according to the restored video data. The data may be displayed.

[0033] In the example of FIG. 3, the terminal devices (310), (320), (330), and (340) are servers, personal computers, The principles underlying this disclosure may be implemented as a personal computer, a smart phone, or the like. The applicability of the present disclosure is not so limited. Laptop Computers, Tablet Computers, Media Players, Wearables It may be implemented in a computer, dedicated video conferencing equipment, etc. The terminal device (310) may include, for example, a wired (wired connection) and / or wireless communication network. ), (320), (330) and (340) for transmitting coded video data between the The communication network (350) 9 represents the number or type of network. , packet-switched channels, and / or other types of channels. Typical networks include telecommunications networks, local area networks, and wide area networks. For the purposes of this discussion, a network (35 0) architecture and topology are incorporated herein by reference unless expressly described herein. may not be important to the operation of

[0034] FIG. 4 illustrates an example of an application of the disclosed subject matter in a video streaming environment. The disclosed subject matter is useful in, for example, video conferencing, digital Digital media including live TV, games, virtual reality, CDs, DVDs, memory sticks, etc. The present invention may be equally applied to other video-enabled applications, including storage of compressed video on media, etc.

[0035] A video streaming system transmits a stream of uncompressed video pictures or images. A video source (401) for creating a video (402), which may include, for example, a digital camera. In one example, a video capture subsystem (413) capable of scanning video pictures. The stream (402) contains samples recorded by the digital camera of the video source 401. The video picture stream (402) is then converted into encoded video data (404) (or To emphasize the high data volume compared to the standard (loaded video bitstream) 4, which includes a video encoder (403) coupled to a video source (401). The video encoder (403) may be configured to process the video by an electronic device (420) including: To enable or implement aspects of the disclosed subject matter as described in more detail below, hardware The encoded video data may include hardware, software, or a combination thereof. The data (404) (or the encoded video bitstream (404)) is a set of uncompressed video pictures. The data volume is shown in thin lines to emphasize its low volume compared to the chat stream (402). The video data is then sent to a streaming server (405) for future use or to a downstream video device (Figure The client subsystems (406) and (407) of FIG. 8) One or more streaming client subsystems such as The server (405) accesses the encoded video data (404) and copies (407) and (409). The client subsystem (406) can acquire, for example, an electronic device (409). The video decoder (410) may include a video decoder (410) for encoding the Decode the input copy (407) of the decoded video data and display (4 12) Rendering onto a display screen or other rendering device (not shown) The video decoder 410 creates an output stream of video pictures (411) that can be , may be configured to perform some or all of the various functions described in this disclosure. In the streaming system, the encoded video data (404), (407), and (409) are ) (e.g., video bitstream) according to a particular video coding / compression standard. Examples of such standards include ITU-T Recommendation H.265. The video coding standard under development is called Versatile Video Coding (VVC). The subject matter disclosed herein may be used in the context of VVC and other video coding standards. It can be used.

[0036] Electronic devices 420 and 430 may include other components (not shown). Note that, for example, the electronic device (420) may include a video decoder (not shown). Alternatively, the electronic device (430) may also include a video encoder (not shown).

[0037] FIG. 5 illustrates a block diagram of a video decoder (510) according to any embodiment of the present disclosure. The video decoder (510) may be included in an electronic device (530). ) may include a receiver (531) (e.g., a receiving circuit). can be used in place of the video decoder (410) in the example of FIG.

[0038] The receiver (531) receives one or more codes to be decoded by the video decoder (510). In the same or another embodiment, the video sequences may be received one at a time. The coded video sequences may be decoded, and each coded video sequence may The decoding of each video sequence is independent of other coded video sequences. A sequence may relate to multiple video frames or images. A video sequence is received from a channel (501), which is a Hardware / software link to storage device that stores the encoded video data, or The receiver (531) may be a streaming source that transmits encoded video data. The encoded video data can be transferred to a respective processing circuit (not shown). together with other data such as input audio data and / or auxiliary data streams. The receiver (531) may receive the coded video sequence from other data. To combat network jitter, a buffer memory (515) may be provided for the receiver. (531) and the entropy decoder / parser (520) (hereafter referred to as "parser (520)" In certain applications, the buffer memory (515) may be disposed between the video decoder (5 10). In other applications, the buffer memory (515) may be implemented as part of a video decoder. In yet other applications, e.g., To counter network jitter, a buffer memory ( (not shown), for example a video decoder (51) for processing playback timing. 0) may have another buffer memory (515). From a controllable storage / forwarding device or over an isosynchronous network When receiving data from the network, the buffer memory (515) may not be necessary or The bandwidth can be made smaller. Best-effort packet networks such as the Internet A buffer memory (515) of sufficient size may be required for the , its size can be relatively large. Such a buffer memory is implemented with an adaptive size. may be implemented external to the video decoder (510) through an operating system or similar component. The present invention may be at least partially implemented in a substrate (not shown).

[0039] The video decoder (510) extracts symbols (521) from the coded video sequence. The parser (520) may be included to recover the symbol categories. Information used to manage the operation of the audio decoder (510) and potentially, as shown in FIG. In addition, the electronic device (530) may or may not be an integral part of the electronic device (5 A rendering device such as a display (512) (e.g., a display screen) that can be coupled to the and information for controlling the rendering device(s). The control information is sent as supplemental enhancement information (SEI) messages or video usability information (VUI) parameters. The parser (520) may be in the form of a meter set fragment (not shown). Parse / entropy decode the coded video sequence received by The coding of the entropy coded video sequence may be performed by It may be according to a coding technique or standard, such as variable length coding, Huffman coding, According to various principles, including coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may be a parser for parsing the coded video sequence. from a video decoder based on at least one parameter corresponding to the subgroup. Extract a set of subgroup parameters for at least one of the subgroups of pixels in the Subgroups include Groups of Pictures (GOP), pictures, tiles, and slides. block, macroblock, coding unit (CU), block, transform unit (TU), The parser (520) can also include the coded From the video sequence, transform coefficients (e.g., Fourier transform coefficients), quantization parameter values , motion vectors, and other information may also be extracted.

[0040] The parser (520) reads the data from the buffer memory (515) to create symbols (521). It is possible to perform entropy decoding / parsing operations on captured video sequences. Cut.

[0041] The reconstruction of the symbols (521) is carried out by the timing of the coded video picture or part thereof. (interpicture and intrapicture, interblock and intrablock, etc.) Depending on the number of different processing or functional units, etc., as well as other factors, The units contained and how they are contained are determined by the parser (520). The subgroup control information parsed from the coded video sequence is Between the parser (520) and the following processing or functional units: Such subgroup control information flow is not shown for the sake of simplicity.

[0042] In addition to the functional blocks already mentioned, the video decoder (510) includes the following: It can be conceptually divided into several functional units, such as: In a working implementation, many of these functional units interact closely with each other, However, various aspects of the disclosed subject matter may be integrated together, at least in part. In order to clearly explain the functionality, the following disclosure adopts a conceptual subdivision into functional units. do.

[0043] The first unit is a scalar / inverse transform unit (551). (551) is a block diagram of the quantized transform coefficients, as well as information indicating what type of inverse transform to use. Control information including block size, quantization coefficients / parameters, quantization scaling matrix, etc. It may receive as symbol(s) (521) from the parser (520). The inverse transformation unit (551) provides sample values ​​that can be input to the aggregator (555). It is possible to output blocks that can be

[0044] In some cases, the output samples of the scaler / inverse transform (551) are intra-coded. A block that is reconstructed using the approximation function, i.e., does not use prediction information from a previously reconstructed picture, , a block that can use prediction information from previously reconstructed parts of the current picture. Such prediction information may relate to the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) can provide , the surrounding blocks that have already been reconstructed and stored in the current picture buffer (558). This information may be used to generate a block of the same size and shape as the block being reconstructed. The current picture buffer (558) may, for example, store a partially reconstructed current picture. and / or buffers the complete reconstructed current picture. In some implementations, for each sample, the intra prediction unit (552) generates The prediction information is converted to output sample information provided by the scaler / inverse transform unit (551). It may be added.

[0045] In other cases, the output samples of the scaler / inverse transform unit (551) are inter-coded. In such cases, the block may be related to a motion-compensated block. The motion compensation prediction unit (553) accesses the reference picture memory (557) to perform the inter-frame motion compensation. The samples used for the block-related prediction can be fetched. After motion compensation of the samples fetched according to symbol (521), these samples The aggregator (555) scales and descales the sampled information to generate the output sample information. The output of the conversion unit (551) can be added to the residual sample A motion compensation prediction unit (553) derives prediction samples from the The address in the reference picture memory (557) to be fetched is, for example, X component, Y component (shift A motion compensated prediction unit in the form of a symbol (521) that may have a reference picture component (time) The motion compensation can be controlled by the motion vectors available in the unit (553). Also, the reference picture memory ( 557), and the motion vector prediction mechanism may include interpolation of sample values ​​fetched from the It may also be associated with, for example,

[0046] The output samples of the aggregator (555) are variously filtered in a loop filter unit (556). Video compression technology can be applied to a wide range of applications, including (coding contained in a coded video sequence (also called a coded video bitstream) It is controlled by the parameters included in the loop as symbols (521) from the parser (520). The filter unit (556) may include in-loop filter techniques that are made available. However, the order of the coded pictures or coded video sequence (decoding order) In addition to responding to meta-information obtained during the decoding of the previous part, It may also respond to loop-filtered sample values, as described in more detail below. As shown, several types of loop filters use loop filter units in various orders. This may be included as part of standard 556.

[0047] The output of the loop filter unit (556) may be output to a rendering device (512). and reference picture memory (557 ) may also be stored in

[0048] A particular coded picture, once fully reconstructed, will be the same as any future interpictures. For example, the current picture can be used as a reference picture for pixel prediction. The coded picture is completely reconstructed and the coded picture is When identified as a picture (e.g., by the parser (520)), the current picture buffer is The reference picture memory (558) can be part of the reference picture memory (557) and can store unused current pictures. The picture buffer is reallocated before starting reconstruction of the next coded picture. It is possible.

[0049] The video decoder (510) is a decoder that uses a predetermined standard, such as ITU-T Rec. H.265. The decoding operation may be performed in accordance with the video compression techniques of The syntax of a video compression technology or standard and a profile documented in the video compression technology In the sense that it complies with both, the coded video sequence is used It may conform to a syntax specified by a video compression technology or standard. Specifically, A profile defines a set of tools that can be used only under that profile, such as video compression. Ability to select specific tools from among all tools available in a technology or standard To comply with the standard, the complexity of the coded video sequence must be increased by the video compression It may be within a range defined by a level of technology or standard. In some cases, the level is , maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., per second Limits the maximum reference picture size, etc. The limits imposed by the IEEE 802.11a standard are determined in some cases by the hypothetical reference decoder (HRD) specification and the Metadata for HRD buffer management signaled in the captured video sequence Thus, it can be further restricted.

[0050] In some exemplary embodiments, the receiver (531) may receive additional The additional data may be received in the form of (one or more) coded The additional data may be included as part of the video sequence that is encoded in the video stream. and / or to more accurately recover the original video data, a video decoder (51 0). The additional data may be used, for example, to estimate time, space, or signal noise. in the form of signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc. could be.

[0051] FIG. 6 illustrates a block diagram of a video encoder (603) according to an exemplary embodiment of the present disclosure. The video encoder (603) may be included in an electronic device (620). The video encoder (603) may further include a receiver (640) (e.g., a transmission circuit). This can be used instead of the video encoder (403) in the example.

[0052] The video encoder (603) receives the video data to be coded by the video encoder (603). A video source (601) (in the example of FIG. 6, an electronic In another example, the video samples may be received from a device (not part of the device 620). The source (601) may be implemented as part of an electronic device (620).

[0053] A video source (601) is a stream to be coded by a video encoder (603). The source video sequence may be encoded in any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, Any color space (e.g. BT.601 Y CrCb, RGB, XYZ...) and any The sampling structure should be appropriate for the RGB signal (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). The media serving system may provide the digital video sample stream. In the system, a video source (601) can store previously prepared video. In a video conferencing system, a video source (601) may store a local image It can be a camera that captures information as a video sequence. It may be presented as a number of separate pictures or images that give the illusion of motion when viewed. The field may be organized as a spatial array of pixels, where each pixel represents the sampling Depending on the structure, color space, etc., it may contain one or more samples. The relationship between pixels and samples can be easily understood by looking at the following. It is in focus.

[0054] According to some exemplary embodiments, the video encoder (603) performs, in real time, or any other time constraint required by the application. The chunks may be coded and compressed into a coded video sequence (643). Enforcing the proper coding rate constitutes one function of the controller (650). In some embodiments, the controller (650) is operatively coupled to other functional units. and may control other functional units as described below. Not shown. Parameters set by the controller 650 include rate Control-related parameters (picture skip, quantizer, lambda value of rate-distortion optimization method) ), picture size, Group of Pictures (GOP) layout, maximum motion vector search The controller 650 may be optimized for a particular system design. The video encoder (603) may be configured to have other suitable functionality associated therewith. can.

[0055] In some exemplary embodiments, the video encoder (603) performs the following steps in the coding loop: As an overly simplified explanation, in one example, the coding The loop includes a source coder (630) (e.g., an input picture to be coded and a and generating symbols, such as a symbol stream, based on a reference picture or pictures. The (local) decoder (633) built into the video encoder (603) The decoder (633) may include an embedded decoder 633 for entropy. Process video streams coded by source coder 630 without coding Even if the code is processed, the symbols are reconstructed and created by the (remote) decoder. (In the video compression technology considered in the subject matter of the disclosure, Any compression between the symbols and the coded video bitstream is lossless. The reconstructed sample stream (sample data) is stored in the reference picture memory ( The symbol stream is then decoded at the decoder's location (local or remote). This leads to bit-accurate results regardless of the number of bits in the reference picture memory (634). The content is also bit-accurate between the local and remote encoders. In other words, the prediction part of the encoder is what the decoder "sees" when using the prediction during decoding. The reference picture samples "see" exactly the same sample values ​​that would otherwise be Synchronization of the data stream (and cases where synchronism cannot be maintained due to channel errors, for example) This basic principle of the resulting drift is used to improve coding quality. It is used for this purpose.

[0056] The operation of the "local" decoder (633) has already been described in detail above in conjunction with FIG. 5. The operation of the decoder 500 may be the same as that of a "remote" decoder, such as a video decoder (510). Simply by reference, however, the symbols are available and the entropy coder (645 ) and encoding of symbols into a coded video sequence by a parser (520). Since the decoding may be lossless, the video decoder includes a buffer memory (515) and a parser (520). The entropy decoding part of the coder (510) is performed in a local decoder (633) within the encoder. Some features may not be fully implemented.

[0057] At this point, all that can be said is that, except for parsing / entropy decoding, which can only exist in the decoder, Any decoder technology will necessarily have substantially the same functionality in the corresponding encoder. Therefore, the disclosed subject matter is a decoder operating system. We may focus on the operation, which is similar to the decoding part of the encoder. The description of the encoder technique is omitted since it is the reverse of the decoder technique which is comprehensively described. Only in certain areas or aspects will a more detailed description of the encoder be provided below. vinegar.

[0058] During operation, in some example implementations, the source coder (630) generates “reference pictures.” One or more previously coded pictures from a video sequence specified as A motion compensated predictive coding method is used to predictively code an input picture by referring to a motion compensated predictive coding method. In this way, the coding engine (632) may 1, which may be selected as a prediction reference (or multiple prediction references) to the input picture. The difference (or residual) in the color channel between the pixel block of the reference picture (or pictures) is copied. The term "residual" and its adjective form "residual difference" are used interchangeably. obtain.

[0059] The local video decoder (633) receives the symbols created by the source coder (630). Based on the coded video data of a picture that can be designated as a reference picture, The operation of the coding engine (632) is advantageously It may be a lossless process. When a video sequence can be decoded by a video decoder (not shown), the reconstructed video sequence usually has some It can be a replica of the source video sequence with some errors. The decoder (633) may perform a decoding process on the reference picture by a video decoder. and store the reconstructed reference picture in the reference picture cache (634). In this way, the video encoder (603) can transmit the A reconstructed reference picture having a common content with a reconstructed reference picture obtained by A copy of the message can be stored locally (without transmission errors).

[0060] The predictor (635) is capable of performing predictive searches for the coding engine (632). That is, for a new picture to be coded, the predictor (635) Candidate reference pixel blocks that can serve as suitable prediction references for a picture Sample data (as a block) or reference picture. Specific data such as motion vectors, block shapes, etc. The reference picture memory (634) can be searched for metadata. ) is calculated for each pixel block against the sample block to find the appropriate prediction reference. In some cases, the search results obtained by the predictor (635) can be used As determined by the result, the input picture is stored in a reference picture memory (634). A picture may have prediction references derived from multiple reference pictures.

[0061] The controller (650) may, for example, control the parameters used to encode the video data. The coding operation of the source coder (630), including the setting of the data and subgroup parameters, You can manage your work.

[0062] The output of all the above functional units is entropy-coded in an entropy coder (645). The entropy coder (645) is a Huffman coder. Reversible coding of symbols according to techniques such as coding, variable length coding, and arithmetic coding Compression converts the symbols produced by the various functional units into a coded video signal. Convert it to an audio sequence.

[0063] The transmitter (640) receives the coded signal generated by the entropy coder (645). Buffering the video sequence and preparing it for transmission over a communication channel (660). The communication channel (660) may include a storage device for storing the encoded video data. The transmitter (640) may be a hardware / software link to a video codec. The coded video data from the data transfer device (603) is transferred to other data to be transmitted, e.g. Coded audio data and / or auxiliary data streams (sources not shown) can be merged with the original (not included).

[0064] The controller (650) can manage the operation of the video encoder (603). During loading, the controller (650) assigns a specific coding to each coded picture. You can assign a coded picture type to each picture. This can affect the coding techniques that can be applied. For example, pictures are often For example, it may be assigned as one of the following picture types:

[0065] Intra pictures (I-pictures) are pictures that use any other picture in the sequence as a prediction source. Some video codecs may be capable of encoding and decoding without requiring any special hardware. For example, different types of Int, including Independent Decoder Refresh ("IDR") pictures. Those skilled in the art will understand these variations of I-pictures and their equivalents. Recognize their uses and characteristics.

[0066] A predicted picture (P-picture) uses at most one moving image to predict the sample values ​​of each block. Using intra prediction or inter prediction using the vector and reference index, It may be capable of being encoded and decoded.

[0067] Bidirectionally predicted pictures (B pictures) use up to two Uses intra or inter prediction using one motion vector and reference index Similarly, multiple predicted images may be encoded and decoded as a single block. Using three or more reference images and associated metadata for rock reconstruction This can be done.

[0068] A source picture is generally made up of multiple sample coding blocks (e.g., each 4×4 The image is spatially subdivided into blocks of 16x16 samples (8x8, 4x8, or 16x16 samples) and The blocks can be coded based on the coding applied to the respective picture. Refer to other (already coded) blocks as determined by the block assignment For example, blocks in an I picture can be coded non-predictively. A new block can be coded or refer to an already coded block of the same picture. The pixels of a P picture can be coded predictively (spatial prediction or intra prediction). Each block predicts the image via spatial prediction, with reference to one previously coded reference picture. Blocks in B pictures may be coded predictively via temporal prediction or as a set of , by spatial prediction with reference to one or two previously coded reference pictures; Alternatively, the source picture or intermediate processed picture may be predictively coded via temporal prediction. A picture may be subdivided into other types of blocks for other purposes. The division of blocks and other types of blocks is done in the same manner as described in more detail below. Methods may or may not be followed.

[0069] The video encoder (603) encodes a video signal in accordance with a predetermined video coding technique such as ITU-T Rec. H.265. In that operation, the video coding operation may be performed according to a technique or standard. The video encoder (603) exploits temporal and spatial redundancies in the input video sequence. Various compression operations can be performed, including predictive coding operations that use The coded video data is stored in a file according to the video coding technology or standard used. Therefore, it can conform to the specified syntax.

[0070] In some exemplary embodiments, the sender (640) may transmit additional The source coder (630) may transmit such data as coded The additional data may be included as part of the video sequence. , other forms of redundant data such as redundant pictures and slices, SEI messages, and VUI parameters It may include set fragments, etc.

[0071] Video may be captured as multiple source pictures (video pictures) in chronological order. Intra-picture prediction (often abbreviated as intra prediction) is the process of predicting the Inter-picture prediction exploits the spatial correlation between pictures, whereas inter-picture prediction exploits the temporal or other correlation between pictures. For example, a particular picture being coded / decoded, called the current picture, is used as the block. A block in the current picture may be divided into blocks that are previously coded in the video. If the reference block is similar to a reference block in a reference picture that is still buffered, the motion vector is A motion vector can be coded by a vector called a motion vector. If multiple reference pictures are used, the reference picture is It is possible to have a third dimension that distinguishes.

[0072] In some exemplary embodiments, bi-prediction techniques may be used for inter-picture prediction. With such bi-prediction techniques, both of the current picture elements in the video are in decoding order. The first reference point (which may be in the past or future in display order) In this case, two reference pictures are used, such as the first reference picture and the second reference picture. The block is a first motion vector pointing to a first reference block in a first reference picture. and a second motion vector pointing to a second reference block in the second reference picture. The block can be coded based on the first and second reference blocks. The combination allows for cooperative prediction.

[0073] In addition, merge mode technology improves coding efficiency in inter-picture prediction. It may also be used to

[0074] According to some exemplary embodiments of the present disclosure, inter-picture prediction and intra-picture prediction are performed. Prediction such as block prediction is performed on a block-by-block basis. For example, For compression, pictures in the image are divided into coding tree units (CTUs), The CTUs in a channel may have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. In general, a CTU consists of three parallel coding tree blocks (CTBs), i.e., one loop Each CTU may contain one or more coding units (CTUs) and two chroma CTBs. For example, a 64x64 pixel CTU can be divided into 64x64 pixel It can be divided into one CU or four CUs of 32x32 pixels. Each of the one or more CUs may be further divided into four CUs of 16x16 pixels. In an embodiment, each CU supports various prediction types, such as inter prediction type and intra prediction type. The CU may be analyzed during encoding to determine which of the CUs to encode. Depending on the predictability and / or spatial predictability, the image is divided into one or more Prediction Units (PUs). In general, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation in coding (encoding / decoding) is performed on a prediction block basis. The division of CUs into PUs (or PBs for different color channels) can be performed in various spatial patterns. The luma PB or chroma PB may include a matrix of sample values ​​(e.g., luma values), such as, for example, 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, and so on.

[0075] FIG. 7 shows a diagram of a video encoder (703) according to another exemplary embodiment of the present disclosure. The video encoder (703) is configured to: Receive a processing block of sample values ​​(e.g., a prediction block) of A coded picture that is part of a loaded video sequence. The exemplary video encoder (703) is configured to (403) can be used instead.

[0076] For example, the video encoder (703) may generate a processing block, such as a prediction block of 8×8 samples. The video encoder (703) then receives a matrix of sample values ​​of the Uses Random Descriptor Optimization (RDO) to determine which processing blocks are best coded using it. The processing determines whether the prediction mode is intra mode, inter mode, or bi-prediction mode. If it is decided that the block is to be coded in intra mode, the video encoder ( 703) uses intra prediction techniques to encode processing blocks into coded pictures. and it is determined that the processing block is coded in inter-mode or bi-predictive mode. If so, the video encoder (703) uses inter-prediction or bi-prediction techniques, respectively. The processing blocks may be encoded into a coded picture using the In an embodiment, as a sub-mode of inter-picture prediction, a motion vector outside the predictor one or more motion vector predictions without the benefit of the coded motion vector components A merge mode derived from the probes may be used. In some other exemplary embodiments, There may be motion vector components applicable to the current block. The da (703) includes a mode decision module, etc., for determining a prediction mode of a processing block. 7. The above-mentioned components may include those not explicitly shown in FIG.

[0077] In the example of FIG. 7, the video encoder (703) is connected to each other as shown in the exemplary configuration of FIG. The inter-encoder (730), the intra-encoder (722), and the residual calculator (72 3), a switch (726), a residual encoder (724), a general-purpose controller (721), and an entry Includes a ropy encoder (725).

[0078] The inter-encoder (730) processes the samples of a current block (e.g., a processing block). , and compares the block with one or more reference blocks (e.g., blocks in the previous and following pictures in the order of display) and the inter prediction information ( (e.g., description of redundant information, motion vectors, merge mode information due to inter-coding techniques) and generating an inter prediction result ( In some examples, the reference pixel may be configured to calculate a prediction block. The decoder 726 may be implemented as a residual decoder 728 in FIG. 7, as described in more detail below. 6. The decoder 620 of FIG. 6 may include a decoder unit 633 that may be used to decode the encoded data. The decoded reference picture is a decoded reference picture that is decoded based on the decoded video information.

[0079] The intra-encoder (722) decodes the samples of the current block (e.g., the processing block). and comparing the block with already coded blocks in the same picture; Generate transformed quantized coefficients and, in some cases, intra prediction information (e.g., one or more The intra-prediction direction information based on the intra-encoding technique is also generated. The intra encoder (722) performs intra prediction based on the intra prediction information and a reference block in the same picture. Based on this, an intra prediction result (eg, a predicted block) may be calculated.

[0080] The general-purpose controller (721) determines general-purpose control data and performs a bidding based on the general-purpose control data. The controller 702 may be configured to control other components of the video encoder 703. The controller (721) determines the prediction mode of the block and switches based on the prediction mode. For example, when the prediction mode is an intra mode, the general The controller (721) controls the switch (726) to select the Select the intra mode result for the image and control the entropy encoder (725) to and selecting intra prediction information and including the intra prediction information in the bitstream. If the description mode of the block is the inter mode, the general-purpose controller (721) (726) to select the inter prediction result for use by the residual calculator (723). , and controls the entropy encoder (725) to select the inter prediction information and This allows for the inclusion of target prediction information in the bitstream.

[0081] The residual calculator (723) calculates the residuals of the received block from the intra-encoder (722) or the inter-encoder (723). - The difference between the prediction result and the block selected from the encoder (730) (residual data) The residual encoder (724) may be configured to encode the residual data to calculate For example, the residual encoder (724) may be configured to convert residual data into transform coefficients. The data may then be configured to transform the data from the spatial domain to the frequency domain to generate transform coefficients. , the transform coefficients are subjected to a quantization process to obtain quantized transform coefficients. In some embodiments, the video encoder (703) also includes a residual decoder (728). 728) is configured to perform an inverse transform and generate decoded residual data. The residual data is then encoded by an intra-encoder (722) and an inter-encoder (730). For example, the inter-encoder (730) may use the decoded residual data A decoded block can be generated based on the inter prediction information and the inter prediction information. The intra encoder (722) decodes the residual data based on the decoded residual data and the intra prediction information. The decoded blocks can generate a decoded picture. The decoded pictures are buffered in a memory circuit (not shown) for proper processing to generate a digital picture. and can be used as a reference picture.

[0082] The entropy encoder (725) generates a bitstream including the encoded blocks. and configured to perform entropy coding. The loopy encoder (725) is configured to include various information in the bitstream. For example, the entropy encoder (725) may include general control data, selected prediction information ( For example, intra-prediction information and inter-prediction information, residual information, and other suitable information may be Either inter-mode or bi-predictive mode can be configured to include the When coding a block in the merge submode, if no residual information is present There is.

[0083] FIG. 8 shows a diagram of an exemplary video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) receives the coding data that is part of the coded video sequence. and receiving a coded picture and decoding the coded picture to obtain a reconstructed picture. In one example, the video decoder (810) is configured to generate the video It may be used in place of the decoder (410).

[0084] In the example of FIG. 8, the video decoder (810) is connected to each other as shown in the exemplary configuration of FIG. The entropy decoder (871), the inter-decoder (880), and the residual decoder ( 873), a reconstruction module (874), and an intra-decoder (872).

[0085] The entropy decoder (871) extracts the coded image from the coded picture. and recovering specific symbols representing syntax elements of which the extracted pictures are composed. Such symbols can be used, for example, to indicate the mode in which the block is coded ( For example, an intra mode, an inter mode, a bi-predictive mode, a merged sub-mode or another sub-mode. (sub-mode), used for prediction by the intra-decoder (872) or inter-decoder (880) Predictive information (e.g., event information) that can identify the particular sample or metadata to be Prediction information (inter-prediction information, intra-prediction information, etc.), e.g. residual information in the form of quantized transform coefficients. In one example, when the prediction mode is an inter mode or a bi-prediction mode, The prediction information is provided to an inter-decoder (880), and the prediction type is an intra-prediction type. In some cases, intra prediction information is provided to the intra decoder (872). It may undergo quantization and is provided to a residual decoder (873).

[0086] The inter decoder (880) receives the inter prediction information and performs a decoding process based on the inter prediction information. The inter prediction unit 100 may be configured to generate inter prediction results using the

[0087] The intra decoder (872) receives the intra prediction information and performs a decoding process based on the intra prediction information. The predictive results may be generated based on the predictions.

[0088] The residual decoder (873) performs inverse quantization to extract the inverse quantized transform coefficients and The residual decoder may be configured to process the coefficients to transform the residual from the frequency domain to the spatial domain. The reader (873) also makes use of specific control information (to include the quantization parameter (QP)). In some cases, this information may be provided by the entropy decoder (871). (Data paths are not shown since there may only be a small amount of control information.)

[0089] The reconstruction module (874) receives, in the spatial domain, the residual The residuals (possibly obtained by an inter-prediction module or an intra-prediction module) The prediction result is combined with the reconstructed image as part of the reconstructed video. The image may be configured to form a reconstructed block that forms part of the captured picture. Other suitable operations, such as deblocking operations, may be performed to improve the quality of the received signal. Please note that:

[0090] Video encoders (403), (603), and (703), and video decoders (410), ( It should be noted that 510) and 810) may be implemented using any suitable technique. In some exemplary embodiments, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) are implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (603), and the video decoders (410), (510), and (810) execute software instructions. It may be implemented using one or more processors for execution.

[0091] Returning to the intra prediction process, a block (e.g., a luma or chroma prediction block, or If it is not further divided into prediction blocks, the samples in the coding block are , to generate a prediction block, the adjacent line, the next adjacent line, or another or multiple lines, or a combination thereof. The residual between the actual block and the predicted block is transformed and then quantized. Various intra prediction modes may be made available, Parameters related to mode selection and other parameters are signaled in the bitstream. Various intra prediction modes may be used, for example, one or more for predicting samples. A prediction sample is selected from multiple line positions, predicting one or more lines. It may relate to the direction and other special intra prediction modes.

[0092] For example, a set of intra prediction modes (interchangeably referred to as "intra modes") may be: A predetermined number of directional intra-prediction modes may be included, as described above in connection with the example implementation of FIG. As such, these intra prediction modes determine the prediction of samples predicted within a particular block. The direction of the out-block samples may correspond to a predetermined number of directions from which the out-block samples are selected. In an exemplary implementation, eight main directional modes can be supported and predefined, corresponding to angles from 45 degrees to 207 degrees relative to the horizontal axis.

[0093] Some other implementations of intra prediction allow for more diverse space in directional textures. To further exploit inter-redundancy, directional intra-mode has finer granularity. For example, the eight-angle implementation above can be expanded to a set of angles, as shown in FIG. , V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED, each of which may be configured to provide eight nominal angles, A predetermined number (e.g., seven) of finer angles may be added to the above. Therefore, a larger total number of intra-modes (e.g., In the example, 56) direction angles can be used for intra prediction. The prediction angle is the nominal intra angle. Each nominal angle can be expressed in degrees + angle delta. There are seven finer angular directions for each nominal angle. In the particular example above, the angle delta could be -3 to 3 times a step size of 3 degrees. .

[0094] In some implementations, instead of or in addition to the directional intra modes described above: A certain number of non-directional intra-prediction modes may also be predefined and made available. Five non-directional intra modes, called low-speed intra prediction modes, are specified. These non-directional intra-mode prediction modes are specifically DC, PAETH, SMOOTH These exemplary modes are sometimes called SMOOTH_H, SMOOTH_V, and SMOOTH_H intra modes. The prediction of samples of a particular block under the directional mode is shown in Figure 10. As an example, FIG. 10 shows a 4×4 block predicted by samples from the upper adjacent line and / or the left adjacent line. 1 illustrates a block 1002. A particular sample 1010 in the block 1002 may correspond to a directly above sample 1004 of the sample 1010 in an above neighboring line of the block 1002, an above-left sample 1006 of the sample 1010 as an intersection of the above neighboring line and the left neighboring line, and a directly left sample 1008 of the sample 1010 in the left neighboring line of the block 1002. In an exemplary DC intra-prediction mode, an average of the left and above neighboring samples 1008 and 1004 may be used as a predictor of the sample 1010. In an exemplary PAETH intra-prediction mode, the above, left, and above-left reference samples 1004, 1008, and 1006 may be fetched, and then whichever value among these three reference samples is closest to (above+left-above-left) may be set as the predictor of the sample 1010. In the example of the SMOOTH_V intra prediction mode, the sample 1010 may be predicted by a quadratic interpolation in the vertical direction of the top-left neighboring sample 1006 and the left neighboring sample 1008. For the example of the SMOOTH_H intra prediction mode, the sample 1010 may be predicted by a quadratic interpolation in the horizontal direction of the top-left neighboring sample 1006 and the top neighboring sample 1004. For the example of the SMOOTH intra prediction mode, the sample 1010 may be predicted by an average of the quadratic interpolations in the vertical and horizontal directions. The above implementations of the non-directional intra modes are provided merely as non-limiting examples. Other implementations may be used. adjacent lines, and other non-directional selection of samples, and specific samples within the predicted block A method of combining prediction samples to predict

[0095] At various coding levels (picture, slice, block, unit, etc.) A particular intra prediction mode by the encoder from the directional or non-directional modes listed above. The selection of the mode may be signaled in the bitstream. In some example implementations, , the exemplary eight nominal direction modes may be signaled first along with the five non-angle smoothed modes (a total of 13 options). The signaled modes are then adjusted to the eight nominal angles. If the mode is one of the intra-degree modes, the selected angle delta is An index is further signaled to indicate the nominal angle that was tracked. In this example implementation, all intra prediction modes are unified for signaling purposes. Together (for example, 56 directional modes plus 5 non-directional modes for a total of 61 intra predictions) Generates a mode) may be indexed.

[0096] In some example implementations, the example 56 or other number of directional intra prediction modes are: Each sample in the block is projected to a reference subsample position and filtered by a 2-tap bilinear filter. This can be implemented using a unified directional predictor that interpolates the reference samples using the

[0097] In some implementations, the method captures the decaying spatial correlation with the fiducials on the edge. To achieve this, an additional filter mode can be designed, called the FILTER INTRA mode. In these modes, the intra-prediction reference samples for some patches in the block are As a result, in-block prediction samples can be used in addition to out-block samples. These modes are, for example, predefined and include at least the luma block (or only the luma block). ) can be made available for intra prediction. A predefined number of filters (e.g., 5) can be used. It is possible to pre-design multiple intramodes, each of which corresponds to a sample in, say, a 4x2 patch. An n-tap filter (e.g., 7 taps) that reflects the correlation between the n neighbors In other words, the weighting coefficients of an n-tap filter are It may depend on the position. Take an example of 8x8 block, 4x2 patch, 7-tap filtering, as shown in Figure 1. As shown in FIG. 1, an 8×8 block 1102 may be divided into eight 4×2 patches. These patches The patches are shown as B0, B1, B2, B3, B4, B5, B6, and B7 in Fig. 11. The patches are shown as R0 to R7 in Fig. 11. The seven neighboring patches can then be used to predict the sample in the current patch. For patch B0, all neighbors may already be reconstructed. However, for other patches, some of the neighbors are in the current block and therefore cannot be reconstructed. may not be built, in which case the predicted values ​​of the nearest neighbors are used as the basis. For example, all the neighbors of patch B7 shown in Figure 11 have not been reconstructed. The predicted samples of the neighboring elements of patch B7 are used instead.

[0098] In some implementations of intra prediction, one color component may be used to predict one or more other color components. The color components can be in any of the YCrCb, RGB, XYZ color spaces, etc. For example, a luma component (e.g., a luma base), called chroma from luma, or CfL, may be used. It is possible to perform prediction of chroma components (e.g., chroma blocks) from the quasi-samples In some example implementations, much of the cross-color prediction is done from luma to chroma. For example, chroma samples within a chroma block must match the The CfL prediction can be modeled as a linear function of the luma samples. This can be done. CfL(α)=α×L AC +DC (1)

[0099] In the formula, L AC represents the AC contribution of the luma component, α represents the parameters of the linear model, and DC , which represent the DC contribution of the chroma components. For example, the AC components are obtained for each sample of a block, The DC component is obtained for the entire block. Specifically, the reconstructed luma samples are The luma values ​​are then subsampled to the luma resolution, and the average luma value (the DC of the luma) is then subtracted from each luma value. The luma AC contributions may then be used in the linear mode of equation (1) to predict the AC values ​​of the chroma components. In order to approximate or predict the chroma AC components from the luma AC contributions, the decoder needs to calculate scaling parameters. Instead, the exemplary CfL implementation determines the parameter α based on the original chroma samples. This allows the decoder to easily This reduces the complexity of the decoder and gives a more accurate prediction. In some example implementations, the chroma content is calculated using intra DC mode. obtain.

[0100] In some example implementations of the baseline, multi-line intra prediction may be used. In these implementations, two or more reference lines are available for selection in intra prediction. In the case of a multi-predictor, the encoder may determine the reference lines used to generate the intra predictor. The reference line index is signaled before the intra prediction mode. The most probable case is when a non-zero reference line index is signaled. Only the higher modes are allowed. Referring to FIG. 15, there are four reference lines (from Reference Line 0 to Reference Line 3) are shown, where each reference line has six segments with a top left reference sample. , i.e., segments A to F (indicated by 1502 to 1512). and F may be padded with the nearest samples from segments B and E, respectively.

[0101] Then, a transform is performed on the residual of either the intra-predicted block or the inter-predicted block. For the purposes of performing the transform, the intra-coded image may be encoded using a quantization technique, and the transform coefficients may then be quantized. Both the inter-coded and inter-coded blocks are Multiple transform blocks (the term "unit transform" is usually used to describe a collection of three color channels) It is sometimes used interchangeably as "conversion unit", e.g., "code The "coding unit" consists of a luma coding block and a chroma coding block. In some implementations, the coding block (or prediction block) may be further divided into A maximum decomposition depth of a coding block may be specified (the term "coding block" is used in place of "coding block"). (This may be used interchangeably with "blocks"). For example, such divisions may not exceed two levels. The division of the predicted block into transform blocks is divided into intra-predicted blocks and inter-predicted blocks. However, in some implementations, Such a partitioning may be similar between intra-predicted and inter-predicted blocks. .

[0102] In some example implementations, for intra-coding blocks, the transform party The transformation may be performed such that all transform blocks have the same size, and the transform blocks are The blocks are coded in raster scan order. An example of block partitioning is shown in Figure 12. Specifically, Figure 12 shows a coding block The lock 1202 is split into the same blocks via a mid-level quadtree split 1204, as shown by 1206. The block size is divided into 16 transform blocks. An exemplary raster scan order is shown by the ordered arrows in FIG.

[0103] In some example implementations, and for inter-coding blocks, the transform unit The partitioning is done recursively up to a given number of partition levels (e.g., two levels). As shown in FIG. 13, the partitioning may be performed at any level for any subpartition. In particular, FIG. 13 shows that block 1302 is a quadtree of four 1304, one of which further performs four second level transformations. While the division of the other subblocks stops after the first level, Here is an example that results in a total of seven transform blocks of different sizes: An exemplary raster scan order is further illustrated by the ordered arrows in FIG. FIG. 13 shows an exemplary implementation of a quadtree division of a square transform block with up to two levels. However, in some production implementations, the transformation partitioning is 1:1 (square), 1:2 / 2:1 and 1:4 / 4:1 conversion block shapes and sizes range from 4x4 to 64x64 In some example implementations, the coding block may be For 64x64 and below, the transform block partitioning is The block can be applied only to the luma component (which is the same as the coding block under that condition). Otherwise, if the width or height of the coding block is greater than 64, the luma Both the coding block and the chroma coding block are min(W,64) Implicitly split into multiples of min(H,64) × min(W,32) × min(H,32) transform blocks. It can be done.

[0104] In some example implementations, as shown in FIG. FIG. 10 further illustrates another alternative exemplary scheme for dividing a prediction block into transform blocks. As shown in Figure 16, instead of using recursive transformation partitioning, we use the coding block A predefined set of partition types is coded according to the conversion type of the block. In the particular example shown in FIG. 16, one of six exemplary partition types may be applied to the block. One may be applied to divide a coding block into various numbers of transform blocks. Such a scheme can be applied to either coding blocks or predictive blocks. In this disclosure, the term "partition type" generally refers to a block (e.g., predicted block or coding block) is partitioned, "Transform partition type", "Prediction block partition type", or "Code It can also refer to a "conversion partition type". For the explanation under "Application Types", the same concept is used as "Coding Block Partition This also applies to "type" and vice versa.

[0105] More specifically, the division scheme of FIG. 16 provides the following for any given transformation type: This method provides up to six partition types for all coding blocks or A prediction block may be assigned a transform type based on, for example, a rate-distortion cost. In the example, the partition type assigned to a coding block or a prediction block is The transformation type of the prediction block or the transformation type of the input block may be determined based on the transformation type of the input block or the prediction block. As shown by the four partition types shown, a particular partition type affects the partitioning of the transform block. It can accommodate a variety of sizes and patterns (or split types). A correspondence between the types of the metric can be predefined. An example correspondence is shown below for the rate-distortion cost: A transform type that can be assigned to a coding block or a prediction block based on the They are shown below with their letter labels.

[0106] ·PARTITION_NONE: Allocate the transformation size equal to the block size.

[0107] PARTITION_SPLIT: Transformation with width 1 / 2 of block size and height 1 / 2 of block size. Allocate size.

[0108] PARTITION_HORZ: A conversion subpart with the same width as the block size and half the height of the block size. Assign a size.

[0109] PARTITION_VERT: A conversion subpart with a width half the block size and a height equal to the block size. Assign a size.

[0110] PARTITION_HORZ4: A conversion subpart with the same width as the block size and 1 / 4 the height of the block size. Assign a size.

[0111] PARTITION_VERT4: A conversion subpart with a width of 1 / 4 of the block size and a height equal to the block size. Assign a size.

[0112] In the above example, all the partition types shown in Figure 16 are This is not a limitation, but merely an example. Some other implementations In this embodiment, the mixed transform block size is the size of the partition for a particular partition type (or pattern). may be used for the selected transform block.

[0113] Then, each of the transformation blocks obtained above may be subjected to a primary transformation. The Mali transform essentially moves the residual in a transform block from the spatial domain to the frequency domain. In some implementations of the actual primary transform, the above example extended coding block is used. To support block partitioning, multiple transformation sizes (one for each dimension of the two dimensions) are used. 4 to 64 points) and transform shape (square; width / height ratio 2:1 / 1:2 , and 4:1 / 1:4 rectangles) are acceptable.

[0114] Turning to the actual primary transformation, in some example implementations, a 2D transformation process The process is carried out by using a hybrid transform kernel (which is, for example, the next An example of a 1-D transform may include the use of a 1-D transform for each element. Conversion kernels may include, but are not limited to: a) 4-point, 8-point, 1 a) 6-point, 32-point, and 64-point DCT-2; b) 4-point, 8-point, and 16-point Asymmetric DST's (DST-4, DST-7) and their flipped versions; c) 4-point, 8-point The choice of transformation kernel used for each dimension is , can be based on a rate-distortion (RD) criterion. For example, DCT-2 and asymmetric The DST's basis functions are listed in Table 1. [Table 1]

[0115] In some example implementations, a hybrid conversion card for a particular primary conversion implementation is used. The availability of the kernel can be based on the transform block size and the prediction mode. Examples of such conversion types are listed in Table 2. For chroma components, the conversion type selection is performed in an implicit manner. In the example, for the intra prediction residual, the transform type, For the inter prediction residual, The conversion type for a romma block follows the conversion type selection of the co-located luma block. Therefore, for the chroma components, the conversion timing is There is no signaling of the type. [Table 2] [Table 3]

[0116] In some implementations, a secondary transform may be performed on the primary transform coefficients. For example, as shown in FIG. 14, in order to further decorrelate the primary transform coefficients, The secondary transform known as LFNST (Low Frequency Non-Separable Transform) is Between primary transform and quantization (at the encoder), and between inverse quantization and inverse primary transform Essentially, LFNST can be applied between the secondary transform and the To proceed, we first need to select some of the primary transform coefficients, e.g., the low-frequency part (hence the transform block). The LFNST can be taken as a "reduced" version of the complete set of primary transform coefficients of the lock. In the example, depending on the transform block size, we have a 4×4 non-separable transform or an 8×8 non-separable transform. For example, small transformation blocks (e.g., min(width, height)<8) can be used. LFNST is applied to the large transformation block (e.g., min(width, height)>8) using 4x4 LFNST. For example, if an 8×8 transform block is subjected to a 4×4 LFNST, , only the low frequency 4×4 portion of the 8×8 primary transform coefficients undergoes a further secondary transform.

[0117] As specifically shown in FIG. 14, the transform block may be 8×8 (or 16×16). Thus, the forward primary transform 1402 of the transform block is an 8×8 (or 16×16) primer The forward LFNS generates a transform coefficient matrix 1404, where each square unit represents a 2×2 (or 4×4) portion. The input to T, for example, does not have to be the entire 8×8 (or 16×16) primary transform coefficients. For example, a 4x4 (or 8x8) LFNST can be used for the secondary transformation. As shown in portion (top left) 1406, the 4×4 (or 8×8) low-frequency Only the wavenumber primary transformation coeffects can be used as input to LFNST. The remaining part of the secondary transform coefficient matrix does not need to undergo a secondary transform. After the secondary transformation, the portion of the primary transformation coeffect that is affected by LFNST is The remaining part that is not affected by LFNST (e.g., the unshaded part of matrix 1404) becomes the retransform coefficient. In some example implementations, the remaining part (the non-transformed part) retains the corresponding primary transform coefficients. The remaining portion not subject to the secondary transform may be set to all zero coefficients.

[0118] An example of the application of the non-separable transform used in LFNST is described below. For this purpose, a 4×4 input block X (e.g., the shaded area of ​​the primary transformation matrix 1404 in FIG. 14) is represents the 4 × 4 low frequency part of the primary transform coefficient block such as min 1406) as follows It can be represented.

number

[0119] This 2D input matrix is ​​first linearized or converted to a vector in the exemplary order:

number

number

[0120] The non-separable transformation of a 4x4 LFNST is then

number

number

number

[0121] The above exemplary LFNST is based on a direct matrix multiplication technique for applying a non-separable transform, and as a result, it is implemented in a single pass without multiple iterations. In some further exemplary Implementations, in order to minimize the computational complexity and memory space requirements for storing transform coefficients, the dimension of the non-separable transform matrix (T) of the 4×4 LFNST example can be further reduced. Such an implementation can be referred to as a reduced non-separable transform (RST). More specifically, the main concept of RST is to map an N (where N is 4×4 = 16 in the above example, but can be equal to 64 for an 8×8 block) dimensional vector to an R dimensional vector in a different space, and N / R (R < N) represents the dimension reduction coefficient. Therefore, instead of an N×N transform matrix, the RST matrix Becomes an R×N matrix as follows. Can be. Such an implementation can be referred to as a reduced non-separable transform (RST). More specifically, the main concept of RST is to map an N (where N is 4×4 = 16 in the above example, but can be equal to 64 for an 8×8 block) dimensional vector to an R dimensional vector in a different space, and N / R (R < N) represents the dimension reduction coefficient. Therefore, instead of an N×N transform matrix, the RST matrix Can be. Such an implementation can be referred to as a reduced non-separable transform (RST). More specifically, the main concept of RST is to map an N (where N is 4×4 = 16 in the above example, but can be equal to 64 for an 8×8 block) dimensional vector to an R dimensional vector in a different space, and N / R (R < N) represents the dimension reduction coefficient. Therefore, instead of an N×N transform matrix, the RST matrix Is to map an N (where N is 4×4 = 16 in the above example, but can be equal to 64 for an 8×8 block) dimensional vector to an R dimensional vector in a different space, and N / R (R < N) represents the dimension reduction coefficient. Therefore, instead of an N×N transform matrix, the RST matrix Becomes an R×N matrix as follows. In Equation 7, the R rows of the transform matrix are the reduced R basis in the N-dimensional space. Therefore, the transform Converts the input vector or the output vector of the reduced R dimension of N dimensions. Therefore, as shown in FIG. 14, the secondary transform coefficient 1408 converted from the primary coefficient 1406 is reduced by a factor of N / R in terms of dimension. The three squares around 1408 in FIG. 14 may be filled with zeros.

Equation

[0122] In Equation 7, the R rows of the transform matrix are the reduced R basis in the N-dimensional space. Therefore, the transform Converts the input vector or the output vector of the reduced R dimension of N dimensions. Therefore, as shown in FIG. 14, the secondary transform coefficient 1408 converted from the primary coefficient 1406 is reduced by a factor of N / R in terms of dimension. The three squares around 1408 in FIG. 14 may be filled with zeros. As shown in FIG. 14, the secondary transform coefficient 1408 converted from the primary coefficient 1406 is reduced by a factor of N / R in terms of dimension. The three squares around 1408 in FIG. 14 may be filled with zeros. Or reduced by N / R. The three squares around 1408 in FIG. 14 may be filled with zeros.

[0123] The inverse transform matrix of an RTS can be the transpose of its forward transform. For a more diverse explanation of the above, in contrast to the 4×4 LFNST, an exemplary reduction factor of 4 and therefore the 64×64 direct non-separable transformation matrix is ​​given by In some implementations, the input primary coefficients are reduced to a 16×64 direct matrix. A portion of may be linearized into the LFNST input vector, rather than the entirety of Only a portion of the input primary transform coefficients may be linearized into the above X vector. So, of the four 4x4 quadrants of the 8x8 primary transform coefficient matrix, the lower right (high frequency coefficients) can be excluded, and only the other three quadrants use a given scan order rather than a 48x1 vector. In such an implementation, the non-separable transformation matrix can be further reduced from 16x64 to 16x48.

[0124] Therefore, the exemplary reduced 48×16 inverse RST matrix is ​​used at the decoder side to reduce the 8×8 Capable of generating the top-left, top-right, and bottom-left 4x4 quadrants of the core (primary) transform coefficients Specifically, instead of a 16x64RST with the same transform set configuration, we use a further reduced 16 When a ×48 RST matrix is ​​applied, the non-separable secondary transformation is the bottom right 4 × 4 block 48 vectorized from three 4x4 quadrant blocks of the 8x8 primary coefficient block excluding matrix elements as input. In such an implementation, the omitted lower right 4×4 plug The primary transform coefficients are ignored in the secondary transform. This further reduced transform is This vector is converted to a 16×1 output vector, which is a 4×4 matrix to fill 1408 in Figure 14. The three squares of secondary transform coefficients surrounding 1408 are zero padded to This is also fine.

[0125] With the help of such a reduction in the dimensions of the RST, we can use memory to store all the LFNST matrices. In the above example, for example, the memory usage is less than that of the implementation without dimensionality reduction. Compared to the previous state, the memory size can be reduced from 10 KB to 8 KB with reasonably little performance degradation.

[0126] In some implementations, to reduce complexity, LFNST is used to LFNST may be further restricted to be applicable only if all coefficients outside the initial transform coefficient portion (e.g., outside 1406 portion of 1404 in FIG. 14) are insignificant. When used, all primary-only transform coefficients (e.g., the primary coefficient matrix in Figure 14) 1404) can be close to zero. Such a limit is imposed by the lowest This allows for the adjustment of LFNST index signaling at the Additional coefficient scans may be required to check for significant coefficients at specific locations when Some implementations use the worst case operation of LFNST (multiplying by 100 per pixel). The above restriction (in terms of computations) can restrict non-separable transforms of 4x4 and 8x8 blocks to 8x16 and 8x48 transforms, respectively. In these cases, for other sizes less than 16, the last significant scan position must be less than 8 if LFNST is applied. For blocks with shapes 4xN and Nx4 and N>8, the above restriction no longer means that LFNST is applied only once to the top-left 4x4 region. Since all primary-only coefficients are 0 when LFNST is applied, the number of operations required for the primary transform is reduced in such cases. From the encoder's point of view, the quantization of coefficients can be simplified when LFNST transforms are tested. Rate-distortion optimized quantization (RDO) can be applied to the first 16 coefficients at most (in scan order). and the remaining coefficients can be set to zero.

[0127] In some exemplary implementations, the available RST kernels are For example, A total of four transformation sets and two non-separable transformation rows per transformation set used in LFNST. There may be sequences (kernels). These kernels may be pre-trained offline. , and is therefore data-driven. The offline trained transform kernels are For use during processing, the data may be stored in memory or in an encoding or decoding device. The selection of the transformation set during the encoding or decoding process can be hard-coded into the The mapping from intra prediction modes to transform sets can be determined by the intra prediction modes. The mapping can be predefined. An example of such a predefined mapping is shown in Table 4. For example, as shown in Table 4, three cross-component linear model (CCLM) modes (INTRA_LT_CCLM, INT RA_T_CCLM or INTRA_L_CCLM) is the current block (i.e., 81<=pred When ModeIntra<=83 is used, transformation set 0 is used for the current chroma block. For each transformation set, the selected non-separable secondary transformation candidates may be selected as follows: It can be further specified by an explicitly signaled LFNST index. For example, the index is synchronized in the bitstream after the transform coefficients, once per intra CU. It can be gunned down. [Table 4]

[0128] LFNST is implemented as follows: In the above implementation, all coefficients outside the first coefficient subgroup or portion are The LFNST index is restricted to be applicable only if it is not significant. The coding may depend on the position of the least significant coefficient. However, it is possible to use text coding, but it does not depend on the intra prediction mode and only the first bin is coded. Furthermore, LFNST supports both intra and inter-text coding. Can be applied to both intra CUs in a slice, as well as both luma and chroma If dual tree is enabled, luma and chroma LFNST indexes are calculated separately. Inter-slice (dual tree disabled) In this case, a single LFNST index is signaled and used for both luma and chroma. It is used.

[0129] In some example implementations, an intra-subpartitioning (ISP) mode is selected. When selected, performance is poor even if RST is applied to all viable partition blocks. Performance improvement is likely to be limited, so LFNST is disabled and the RST index is signaled. In addition, disabling RST for ISP prediction residuals reduces the coding complexity. In some further implementations, multiple linear regression intra-prediction ( When MIP) mode is selected, LFNST is also disabled and the RST index is not signaled. It is not necessary.

[0130] Due to the existing maximum transform size limit (e.g. 64x64), large CUs ( or any other predefined size representing the maximum transform block size) is implicitly split (e.g. Considering that the LFNST index search is performed on a certain number of decoded packets, Data buffering can be increased by a factor of four for the pipeline stage. Thus, in some implementations, the maximum size allowed for an LFNST is limited to, for example, 64x64. In some implementations, LFNST is only valid with DCT2 as the primary transform. It can be.

[0131] In some other implementations, for example, with three kernels in each set, e.g. We define 12 sets of secondary transforms to perform the Intra Secondary Transform (IST) on the luma component. ) is provided. An intra-mode dependent index is used for transformation set selection. The kernel selection within the set may be based on signaled syntax elements. When either DCT2 or ADST is used as both the horizontal and vertical primary transforms, In some implementations, a 4×4 separation may be enabled according to the block size. You can choose between a non-separable transform or an 8x8 non-separable transform. _height) < 8, 4x4 IST can be chosen. For larger blocks, 8 ×8IST can be used. Here, tx_width&tx_height are the conversion blocks. The input to the IST is a set of low frequency primaries in a zigzag scan order. It may be a conversion coefficient.

[0132] Various transformations in the video coding or decoding process, e.g., within a residual block The primary transformation of the sample or the secondary transformation of the block of the primary transformation coefficient process If only separable conversion methods are used, one of the conversions will be at a 45 degree angle (e.g., horizontally). Oriented texture patterns, such as edges that are substantially away from the vertical As mentioned above, some In an exemplary implementation, one or more separations are performed for the secondary transformation of the primary transform coefficients. Impossible conversion designs can be used.

[0133] Transformation block partitioning and transformation type applied to the partitioned transformation blocks For example, a particular transformation type may be related to a particular partition type. For example, the transform partitioning illustrated in FIG. 16 and described above may be more suitable. The methodology uses non-recursive partition types as compared to recursive partitions such as the partition previously described in FIG. All available conversion types are displayed in the For transformation blocks partitioned under the transformation partition type (e.g., transformation partition type in Figure 16), If allowed, the encoder can select which transformation party to use to obtain the transformation block. What partitioning type to use and how to allocate it for each partitioned transformation block When deciding which transformation type to use, you must perform optimization over a large parameter space. In practice, a certain set of conversion types will generally be required for a particular type. Some conversion types may be more appropriate for some partition types than others. Various implementations of the transform partition type and the transform type may take into account the interaction between the transform partition type and the transform type. Consider and use the following to restrict the types of conversions allowed for a particular partition type: Similarly, we have arrived at a method to restrict the partition types allowed for a given conversion type. Such an implementation is particularly suited to non-recursive transformation partitioning. In the situation where the transform partition pattern and transform type of each divided transform block are used, When determining the loop selection, it may allow a reduction in the optimization space for the encoder.

[0134] These example implementations may be used separately or in any order or manner. In the above and following discussion, "coded blocks" may be used. The terms "picture unit," "coding block," and the like refer to a picture unit on which prediction or transformation is performed. The coding block may be used to refer to a luma coding block. In some situations, the coding block may be a chroma coding block. The coded / coded block may refer to a predicted block. The term refers to the width or height of a coding block, or the maximum of the width and height, or the width and Minimum height, or area size (width*height), or aspect ratio (width:height, or height It is used to refer to the width of a square.

[0135] Multiple candidate primary conversion types

[0136] In one embodiment, there may be multiple candidate primary transformation types for a block. The primary transform task may include the transform blocks resulting from the partition. The selection and / or signaling of the type is based on a predefined set of transformation partition types. A predefined set of transformation partition types can be used to limit the available transformations. A larger set of exchangeable partition types (e.g., the partition type flag in Figure 16) In other words, the transformation partition type of a block can be a subset of the The primary partition is created only if the partition belongs to a predefined set of conversion partition types. A transformation selection is made and signaled. Instead, e.g., transformation partition If the type is anything else, it is used as the default rather than being selected and signaled. The default conversion type may be used.

[0137] In one implementation, the primary transform type is a discrete cosine transform (DCT) type 1 to DCT Asymmetric Discrete Sine Transform (ADST) Type 8; Discrete Sine Transform (DST) Type 1 to DST Type 8 At least one of the following: Line Graph Transformation (LGT); or Karhunen-Loeve Transformation (KLT) It may include one.

[0138] In one implementation, the predefined set of transformation partition types is, for example, Among the various conversion partition types, only PARTITION_NONE is included, i.e. The block size is equal to the prediction block (or coding block) size. The primary transformation type is only available if the transformation type belongs to this predefined set. can be selected and / or signaled.

[0139] In one implementation, the number of partitions is also determined by a predefined number of partitions for the conversion partition type. For example, the number of conversions for a particular conversion partition type may be considered to determine the selected set. Primary transformation type only if the number of partitions is below a predefined threshold A predefined set of transformation partition types from which the In one implementation, the predefined threshold is an integer between 1 and 16. This is also fine.

[0140] Multiple possible secondary transformation types

[0141] In one embodiment, there may be multiple candidate secondary transformation types for a block. The secondary transform task may include the transform blocks resulting from the partition. The selection and / or signaling of the type is based on a predefined set of transformation partition types. A predefined set of conversion partition types is available. A larger set of possible conversion partition types (e.g., the conversion partition types in Figure 16) In other words, the transformation participants of a block can be a subset of the full set of transformation types. A partition type is converted to a partition only if it belongs to a predefined set of partition types. Instead, the choice of the conversion party is made and signaled. If the partition type is of another type, it is not selected and signaled. A default secondary transformation type may be used, or a secondary transformation may be performed. It's not necessary.

[0142] In one implementation, the secondary transform type may include a KLT, which is a transform with a different kernel. It may be configured.

[0143] In one implementation, the predefined set of transformation partition types is, for example, Of the various conversion partition types, only PARTITION_NONE may be included, i.e. The transformation block size is equal to the prediction block (or coding block) size. Therefore, the secondary conversion type is only available if the conversion type belongs to this predefined set. The type can be selected and / or signaled.

[0144] In one implementation, the number of partitions is also determined by a predefined number of partitions for the conversion partition type. For example, the number of conversions for a particular conversion partition type may be considered to determine the selected set. Secondary transformation type only if the number of partitions is below a predefined threshold A predefined set of transformation partition types from which the In one implementation, the predefined threshold is an integer between 1 and 16. This is also fine.

[0145] In one implementation, the selection and / or signaling of the secondary conversion type is performed by the conversion party. The combination may be based on a combination of the partition type and the primary transformation type. a conversion partition type in a predefined set of conversion partition types; and a primary transform type in a predefined set of transform types. For example, The secondary transformation type is a transformation partition type of PARTITION_NONE and a block Select only if the primary transformation type used in the block is DCT or ADST and / or Instead, the conversion partition type may need to be signaled, e.g. If it is of another type, it will be used as the default secondary instead of being selected and signaled. A secondary transformation type may be used, or no secondary transformation may be performed.

[0146] Transformation-related signaling

[0147] In this disclosure, the order of transformation-related syntax elements / parameters is taken into consideration in signaling. Various signaling mechanisms are disclosed that aim to increase efficiency.

[0148] In one embodiment, the conversion partition type information is a primary / secondary conversion type. The primary / secondary transform type selection may be signaled before the transform selection information. The conversion partition is a predefined set of conversion partition types, e.g. PARTIT It needs to be signaled only when it belongs to ION_NONE. If the partition does not belong to the predefined set of conversion partition types, the primary The primary / secondary transform type selection may not need to be signaled. Instead, The primary and secondary conversion types are introduced as predefined default conversion types. It may be issued.

[0149] In one embodiment, the primary / secondary conversion type selection information is a conversion partition. It may be signaled before the type information. In this case, the selection of the conversion partition type and The signaling and / or the signaling may depend on the primary / secondary conversion type selection information.

[0150] In one implementation, the primary conversion type belongs to a predefined set of conversion types. Only in this case, the conversion partition type information needs to be signaled. Otherwise, the conversion partition type information may not need to be signaled. As an example, the predefined set of transformation types is DCT type 1 through DCT type 8, ADST, These may include, but are not limited to, DST Type 1 to DST Type 8, LGT, and KLT.

[0151] In one implementation, the secondary transformation type belongs to a predefined set of transformation types. Transform partition type information may need to be signaled only if The predefined set of transformation types includes, but is not limited to, the predefined KLT interfaces. It is possible to include a specific KLT that has a kernel associated with it. If the secondary transformation type does not belong to the predefined set of transformation types, Partition type information may not need to be signaled. Instead, the conversion The partition type information is a predefined default conversion partition, such as PARTITION_NONE. It may be derived as a partition type.

[0152] 17 illustrates an example method 1700 for decoding video data. The method 1700 includes the following steps: receiving a coded video bitstream for a data block at step 1710; and extracting from the coded video bitstream a transform partition type associated with the data block at step 1720. Step 1730, the conversion partition type converts the data block into a conversion block. A conversion partition type pre-defined partition pattern for dividing the blocks into blocks. In response to belonging to a subset of the defined set, the coded video bits are Transformation blocks separated from data blocks as signaled in the data stream. extracting a transformation type of a transformation associated with the transformation, The transformation blocks are organized according to the steps and transformation types that belong to a first predefined set of transformation types. and performing an inverse transformation on the lock.

[0153] In embodiments of the present disclosure, any step and / or action may be performed in any quantity or amount as desired. Two or more steps and / or actions may be combined or arranged in a sequence. can be executed in parallel.

[0154] The embodiments of the present disclosure may be used separately or combined in any order. Additionally, each of the methods (or embodiments), the encoder, and the decoder may include a processing circuit (e.g., , one or more processors, or one or more integrated circuits) In one example, the one or more processors may be stored in a non-transitory computer readable medium. The embodiment of the present disclosure is applicable to the luma block or the chroma block. may be used.

[0155] The techniques described above involve the use of computer-readable media physically stored on one or more computer-readable media. The present invention may be implemented as computer software using computer readable instructions. For example, FIG. 18 illustrates a computer system suitable for implementing certain embodiments of the disclosed subject matter. This indicates the year (1800).

[0156] Computer software is software that runs on one or more computer central processing units (CPUs). The processing unit (GPU) and graphics processing unit (GPU) are used to Contains instructions that can be executed directly by the processor or through interpretation and execution of microcode, etc. Assembly, compiling, linking, or similar mechanisms to create the code It may be coded using any suitable machine code or computer language that can do.

[0157] The instructions may be stored in, for example, a personal computer, a tablet computer, a server, a smart Various types of computers including smartphones, game consoles, Internet of Things devices, etc. It may be implemented on a computer or components thereof.

[0158] The components shown in FIG. 18 for computer system 1800 are illustrative in nature. and the scope of use or functionality of computer software implementing embodiments of the present disclosure. The configuration of components is not intended to imply any limitation on the Any one of the components shown in the exemplary embodiment of the computer system (1800) Neither should it be construed as having any dependency or requirement relating to combinations.

[0159] The computer system (1800) includes a specific human interface input device. Such a human interface input device may be, for example, a tactile input (keypad strokes, swipes, data glove movements, etc.), voice input (voice, clapping, etc.), One or more human users may be able to sense the surroundings through sensory input (such as gestures), olfactory input (not shown), and / or other sensory inputs. The human interface device can respond to voice (speech, music, ambient sounds, etc.), images (scanned images, photographic images obtained from still image cameras, etc. ), video (two-dimensional video, three-dimensional video including stereoscopic video, etc.), etc. It is also possible to capture specific media that are not necessarily directly related to the intended input.

[0160] The input human interface devices include a keyboard (1801), a mouse (1802), Rack pad (1803), touch screen (1810), data glove (not shown), One of the following: istik (1805), microphone (1806), scanner (1807), camera (1808) The device may include one or more (one of each is shown).

[0161] The computer system (1800) also includes a particular human interface output device. Such human interface output devices may include, for example, haptic output, It may stimulate one or more of the human user's senses through sound, light, and smell / taste. Human interface output devices such as haptic output devices (e.g., touch The screen (1810), data glove (not shown), or joystick (1805) A haptic feedback device that provides haptic feedback but does not function as an input device. A speaker (1809), a headphone (not shown), and an audio output device (1810). Visual output devices (CRT screens, LCD screens, plasma screens, etc.) OLED screens (with or without touchscreen input capability, respectively) Some of these may provide 2D visual output or a sensory feedback function. It may be possible to output more than three dimensions by means of telegraphic output or other means. ), virtual reality glasses (not shown), a holographic display The equipment may include a power supply, power source ...

[0162] The computer system (1800) also includes human-accessible storage devices and their associated Optical media including CD / DVD ROM / RW (1820) with media such as CD / DVD ( 1821), thumb drive (1822), removable hard drive or solid state drive Eve (1823), legacy magnetic media such as tapes and floppy disks (not shown), security dongle (not shown) and other dedicated ROM / ASIC / PLD-based devices It is possible.

[0163] Those skilled in the art will also be familiar with the concepts of "computer-readable media" as used in connection with the presently disclosed subject matter. It is understood that the term "transmission media" does not encompass transmission media, carrier waves, or other transitory signals. It should be.

[0164] The computer system (1800) also includes a network (1855) for communicating with one or more communication networks (1855). The network may include, for example, a wireless, wired, optical, The network can be further classified into local, wide area, metropolitan, vehicular and industrial, It can be real-time, delay tolerant, etc. Examples of networks include Ethernet , Local Area Networks such as Wi-Fi, GSM, 3G, 4G, 5G, LTE, etc. Television, including local networks, cable, satellite, and terrestrial broadcast television Wired or wireless wide area digital networks, including CANbus, vehicle and industrial. A particular network is usually connected to a USB port on a computer system (1800), for example. Any external network attached to a specific general purpose data port or peripheral bus (1849) An interface adapter is required, and other networks are typically The core of the computer system (1800) is integrated by attaching it to the system bus ( For example, an Ethernet interface to a PC computer system or a smartphone cellular network interfaces to computer systems. The computer system (1800) can communicate with other entities using any of the following: Such communications may be one-way, receive only (e.g., television broadcast), or one-way, transmit only (e.g., CANbus to a specific CANbus device), or bidirectional, e.g., local or wide area Communication to other computer systems using real digital networks. The protocols and protocol stacks, as explained above, are It can be used on both the IEEE 802.11b and network interfaces.

[0165] the human interface device, the human accessible storage device, and The network interface is connected to the core (1840) of the computer system (1800). It is possible.

[0166] The core (1840) may include one or more central processing units (CPUs) (1841), a graphics processing unit (GPU), (GPU)(1842), Field Programmable Gate Area (FPGA)(1843) Dedicated programmable processing units for specific tasks, hardware accelerators (1844 ), a graphics adapter (1850), etc. These devices can read Read-only memory (ROM) (1845), random access memory (1846), user-accessible Internal mass storage devices such as internal hard drives, SSDs, etc. (1847) cannot be used with the system. Some computer systems can be connected via a USB bus (1848). A system bus in the form of one or more physical plugs to allow expansion by PUs, etc. The peripheral devices can access the core's system bus (1848). The devices may be connected directly or through a peripheral bus (1849). The peripheral bus architecture allows the connection of a graphics adapter (1850) to a peripheral bus (1810). The architecture includes PCI, USB, etc.

[0167] CPU (1841), GPU (1842), FPGA (1843), and accelerator (1844) are combined The computer can execute the specific instructions that, when combined with the computer program, constitute the computer code. The code can be stored in ROM (1845) or RAM (1846). The transition data is stored in RAM (1846). Alternatively, permanent data may be stored, for example, on an internal mass storage device (1847). High speed storage and retrieval of any memory device is achieved by one or more CPUs (1841), GPUs (1842) , mass storage device (1847), ROM (1845), RAM (1846), etc. This can be made possible through the use of cache memory.

[0168] The computer readable medium may include computer code for performing various computer-implemented operations. The media and computer code may include may be designed and constructed by a person skilled in the art of computer software technology. The type of the filter may be of any known and available type.

[0169] As a non-limiting example, a computer system (1800) having an architecture, particularly A (1840) is a processor (including CPU, GPU, FPGA, accelerator, etc.) The processor may include software embodied in one or more tangible computer readable media. Such a computer-readable medium can provide a function as a result of executing the program. The user-accessible mass storage devices introduced above, as well as the core internal mass storage devices, Related to certain storage devices of a non-transient nature, such as memory (1847) and ROM (1845) The software implementing the various embodiments of the present disclosure may be stored in a hard disk drive or a hard disk drive attached to such a hard disk drive. The computer-readable medium may be stored in such a device and executed by the core (1840). , which may include one or more memory devices or chips, depending on the particular need. The software is responsible for the cores (1840), specifically the processors (CPU, GPU, and FP GA, etc.) to define the data structure stored in the RAM (1846), and modifying such data structures according to a process defined by the software; Performing a particular process or a particular part of a particular process described herein Additionally or alternatively, the computer system may be The software is used to execute a particular process or a particular part of a particular process. may operate in place of or in conjunction with software, hardwired into circuitry, or otherwise Providing functionality as a result of logic embodied in a way (e.g., Accelerators (1844)) References to software include logic, where appropriate. Where appropriate, reference to a computer-readable medium may also refer to a A circuit (such as an integrated circuit (IC)) that stores software for The present disclosure may include a circuit embodying logic for It encompasses any suitable combination of hardware and software.

[0170] While this disclosure has described several exemplary embodiments, modifications and variations that fall within the scope of this disclosure are possible. , substitutions, and various alternative equivalents. Although not shown or described, it embodies the principles of the present disclosure and therefore It will be appreciated that numerous systems and methods can be devised that are within the spirit and scope of the present invention. Appendix A: Acronyms JEM: Joint Exploration Model VVC: Versatile Video Coding BMS: Benchmark Set MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Supplemental Enhancement Information VUI: Video Usability Information GOP: Group of Pictures TU: conversion unit PU: Prediction unit CTU: Coding Tree Unit CTB: coding tree block PB: Predicted block HRD: Hypothetical Reference Decoder SNR: Signal to Noise Ratio CPU: Central Processing Unit GPU: Graphics Processing Unit CRT: cathode ray tube LCD: Liquid crystal display OLED: Organic Light Emitting Diode CD:Compact Disc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile Communications LTE: Long-Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Area SSD: Solid State Drive IC: Integrated Circuit HDR: High Dynamic Range SDR: Standard Dynamic Range JVET: Joint Video Exploration Team MPM: Most Probable Mode WAIP: Wide-angle Intra Prediction CU: coding unit PU: Prediction unit TU: conversion unit CTU: Coding Tree Unit PDPC: Position-dependent prediction combination ISP: Intra Subpartition SPS: Sequence Parameter Settings PPS: Picture Parameter Set APS: Adaptive Parameter Set VPS: Video Parameter Set DPS: Decoding Parameter Set ALF: Adaptive Loop Filter SAO: Sample Adaptive Offset CC-ALF: Cross-component adaptive loop filter CDEF: Constrained Directional Enhancement Filter CCSO: Cross component sample offset LSO: Local Sample Offset LR: Loop recovery filter AV1:AOMedia Video1 AV2:AOMedia Video2 [Explanation of symbols]

[0171] 101 Samples 102 Arrow 103 Arrow 104 Square Block 201 Block 300 Communication Systems 310 Terminal Equipment 320 Terminal Equipment 330 Terminal Equipment 350 Network 400 Communication Systems 401 Video Source 402 Stream 403 Video Encoder 404 Video Data, Video Bitstream 405 Streaming Server 406 Client Subsystem 407 Input copy, video data 410 Video Decoder 411 Output Stream 412 Display 413 Video Capture Subsystem 420 Electronic equipment 430 Electronic equipment 501 Channel 510 Video Decoder 512 Rendering devices, displays 515 Buffer Memory 520 Parser 521 Symbols 530 Electronic equipment 531 Receiver 551 Reverse conversion unit 552 Intra-picture prediction unit 553 Motion Compensation Prediction Unit 555 Aggregator 556 Loop Filter Unit 557 Reference Picture Memory 558 Picture Buffer 601 Video Sources 603 Video Coder, Video Encoder 620 Electronic equipment 630 Source Coder 632 Coding Engine 633 Local Video Decoder, Decoding Unit 634 Reference Picture Cache, Reference Picture Memory 635 Predictor 640 Transmitter 643 Video Sequences 645 Entropy Coder 650 Controller 660 Communication Channels 703 Video Encoder 721 General-purpose controller 722 Intra Encoder 723 Residual Calculator 724 Residual Encoder 725 Entropy Encoder 726 Switch 728 Residual Decoder 730 InterEncoder 810 Video Decoder 871 Entropy Decoder 872 Intra Decoder 873 Residual Decoder 874 Reconstruction Module 880 Interdecoder 1002 Block 1004 Top Sample 1006 Top left sample 1008 Left Sample 1010 Sample Block 1102 1202 coded blocks 1204 Quadtree splitting 1302 Block 1304 Quadtree Subblock 1402 Forward Primary Transformation 1404 Primary transformation coefficient matrix 1406 Primary coefficients, shaded area 1408 Secondary Conversion Factors 1800 Computer Systems 1801 Keyboard 1802 Mouse 1803 Trackpad 1805 Joystick 1806 Mike 1807 Scanner 1808 Camera 1809 Audio Output Device Speaker 1810 Touch Screen 1821 Optical media 1822 Thumb Drive 1823 Solid State Drive 1840 Core 1843 Field Programmable Gate Area (FPGA) 1844 Hardware Accelerator 1845 Read-Only Memory (ROM) 1846 Random Access Memory 1847 Core Internal Mass Storage 1848 System Bus 1849 Peripheral Bus 1850 Graphics Adapter 1854 Interface 1855 Communication Network

Claims

1. 1. A method for encoding video data, comprising the steps of: determining a transformation partition type associated with a data block of the video data; in response to the transform partition type belonging to a subset of a predefined set of transform partition types, each transform partition type specifying a partitioning pattern for dividing the data block into transform blocks, and a number of transform partitions associated with each transform partition type in the subset of the predefined set of transform partition types is less than or equal to a predefined threshold; determining a transformation type of a transformation associated with a transformation block split from the data block, the transformation type belonging to a first predefined set of transformation types; performing a transform on the transform block according to the transform type; in response to the transformation partition type not belonging to the subset of the predefined set of transformation partition types; performing a transformation on the transformation block according to a default transformation type; A method comprising:

2. in response to the transformation partition type not belonging to the subset of the predefined set of transformation partition types; The method of claim 1 , further comprising determining the transform type associated with the transform block to be a predefined default transform type.

3. The transform is a primary transform, and the first predefined set of transform types is: Discrete cosine transform (DCT) type 1 to DCT type 8, Asymmetric Discrete Sine Transform (ADST), Discrete Sine Transform (DST) Type 1 to DST Type 8, Line graph transformation (LGT), and 3. The method of claim 1 or 2, comprising a Karhunen-Loeve transformation (KLT).

4. The method of claim 1 , wherein the subset of the predefined set of transform partition types consists of PARTITION_NONE for no transform block partitioning.

5. The method of claim 1 , wherein the predefined threshold comprises an integer between 1 and 16, inclusive.

6. The method of claim 1 , wherein the transform is a secondary transform and the first predefined set of transform types includes a KLT.

7. The method of claim 1 , wherein the subset of the predefined set of transform partition types consists of PARTITION_NONE for no transform block partitioning.

8. The method of claim 1 , wherein a number of transform partitions associated with each transform partition type in the subset of the predefined set of transform partition types is less than or equal to a predefined threshold.

9. The method of claim 1 , wherein the transform type further indicates that a type of a primary transform associated with the transform block belongs to a second predefined set of transform types.

10. 10. The method of claim 9, wherein the subset of the predefined set of transform partition types consists of PARTITION_NONE for no transform block partitioning, and the second predefined set of transform types consists of DCT and ADST.

11. 1. A method for encoding video data, comprising the steps of: determining a transformation partition type associated with a data block of the video data; determining a transform type associated with a transform block of the data block in response to the transform partition type belonging to a subset of a predefined set of transform partition types and a number of transform partitions associated with each transform partition type in the subset of the predefined set of transform partition types being less than or equal to a predefined threshold; identifying the conversion type of the data block by default in response to the conversion partition type not belonging to the predefined set of conversion partition types; A method comprising:

12. The method of claim 11 , wherein the subset of the predefined set of transform partition types consists of PARTITION_NONE for no transform block partitioning.

13. The method of claim 11 or 12, wherein the transformation type comprises a primary transformation type or a secondary transformation type.

14. 14. An apparatus comprising circuitry configured to perform the method of any one of claims 1 to 13.

15. A computer program product causing one or more processors to carry out the method according to any one of claims 1 to 13.

Images