Method, apparatus and storage medium for video coding

Abstract

The present application includes a method and an apparatus comprising computer code configured to cause one or more processors to perform operations of receiving video data in an open media alliance video 1 (AV1) format, the video data comprising data of at least two blocks of chroma prediction residual signals; performing a transform between at least one signal block and the blocks of chroma prediction residual signals, the at least one signal block having a size less than or equal to a combination of the blocks of chroma prediction residual signals; and decoding the video data based on an output of the transform, the output of the transform comprising the at least one signal block having the size less than or equal to the combination of the blocks of chroma prediction residual signals.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 000,109, filed March 26, 2020, and U.S. Application No. 17 / 083,662, filed October 29, 2020, the entire contents of which are expressly incorporated herein by reference. Technical Field

[0003] This application generally relates to the field of data processing, and more specifically, to video encoding and / or decoding. Background Technology

[0004] AOMedia Video1 (AV1) is an open video coding format designed for transmitting video over the Internet. AV1 was developed by the Alliance for Open Media (AOMedia) as a successor to VP9. AOMedia is a consortium founded in 2015 that includes semiconductor companies, video-on-demand providers, video content producers, software development companies, and web browser vendors. AV1 has a total of 56 directional angles, of which 8 are nominal angles, and the rest are specified as delta increments of the nominal angle.

[0005] Many components of the AV1 project stem from previous research work by consortium members. Individual contributors began experimenting with the technology platform several years ago: Xiph / Mozilla's Daala released code in 2010, Google's experimental VP9 evolution project VP10 was released on September 12, 2014, and Cisco's Thor released it on August 11, 2015. Building upon the VP9 code, AV1 incorporates additional technologies, some of which were developed within these experimental formats. The first version 0.1.0 of the AV1 reference codec was released on April 7, 2016. The consortium announced the release of the AV1 bitstream specification, along with software-based reference encoders and decoders, on March 28, 2018. Validation version 1.0.0 of the specification was released on June 25, 2018. Validation version 1.0.0 of the specification and Errata 1 were released on January 8, 2019. The AV1 bitstream specification includes a reference video codec.

[0006] The Video Coding Experts Group (ITU-T VCEG) (Q6 / 16) of the Telecommunication Standardization Sector of International Telecommunication Union (ITU-T) and the Moving Picture Experts Group (ISO / IEC MPEG) (JTC 1 / SC 29 / WG11) of International Organization for Standardization (ISO) published the first version of H.265 / HEVC (High Efficiency Video Coding) standard in 2013, the second version of H.265 / HEVC standard in 2014, the third version of H.265 / HEVC standard in 2015, and the fourth version of H.265 / HEVC standard in 2016. Since then, they have been studying the potential need for standardization of future video coding technology that can significantly outperform HEVC in compression capability. In October 2017, they published a joint call for proposal (CfP) on video compression beyond HEVC. As of February 15, 2018, there were 22 CfP responses for standard dynamic range (SDR), 12 CfP responses for high dynamic range (HDR), and 12 CfP responses for 360 video categories. In April 2018, all the received CfP responses were evaluated at the 122nd MPEG / 10th JVET (Joint Video Exploration Team) meeting. After careful evaluation, the JVET officially started the standardization of the next generation video coding beyond HEVC, i.e., so-called Versatile Video Coding (VVC).

[0007] In AV1, prediction residual signals are generated for chroma channels (e.g., Cb and Cr) through various means and applied in intra and inter coding scenarios, respectively. Since the same set of prediction methods are applied to Cb and Cr channels in most cases, the resulting residual signals from different chroma channels tend to be related in a certain classifiable way. Note that AV1 does not have coding tools to exploit this redundancy for better compression efficiency. On the other hand, even though the current VVC design does provide a way to exploit this trend, it is limited in the sense of exploiting the correlation between only two co-located chroma samples. SUMMARY

[0008] This application presents a set of advanced video coding techniques. More specifically, this application discloses a transform scheme that jointly codes residuals from multiple color components (e.g., residuals from two chroma components).

[0009] This application includes a method and apparatus, the apparatus comprising a memory configured to store computer program code and one or more processors configured to access and execute the computer program code according to its instructions. The computer program includes: receiving code configured to cause the at least one processor to receive AV1 format video data, the video data including data of at least two chroma prediction residual signal blocks; execution code configured to cause the at least one processor to perform a transformation between at least one signal block and the chroma prediction residual signal blocks, the size of the at least one signal block being less than or equal to a combination of the chroma prediction residual signal blocks; and decoding code configured to cause the at least one processor to decode the video data based on the output of the transformation, the output of the transformation including the at least one signal block whose size is less than or equal to the combination of the chroma prediction residual signal blocks.

[0010] According to an exemplary embodiment, the chromaticity prediction residual signal block includes an N1xN2 Cb residual block and an N1xN2 Cr residual block.

[0011] According to an exemplary embodiment, the transformation includes: dequantizing the signal-notified indexes of the coefficients of the at least one signal block, the size of the at least one signal block being less than or equal to the combination of the chroma prediction residual signal blocks; and transforming the dequantized signal-notified indexes by multiplying a 2xN1xN2 matrix by an LxM matrix. 2xN1xN2 is the size of the combination of the chroma prediction residual blocks, and LxM is the size of the combination of the chroma prediction residual blocks being less than or equal to the size of the combination of the chroma prediction residual blocks.

[0012] According to an exemplary embodiment, the result of transforming the dequantized signal-notified index by multiplying it by a 2xN1xN2 LxM matrix includes a reconstructed 2xN1xN2 vector; and decoding the video data includes reconstructing N1xN2 Cb residual blocks and N1xN2 Cr residual blocks based on the reconstructed 2xN1xN2 vector.

[0013] According to an exemplary embodiment, the at least one signal block includes the interleaving of the at least two chroma prediction residual signal blocks.

[0014] According to an exemplary embodiment, the at least one signal block comprises a 2xN1xN2 three-dimensional (3-D) cube.

[0015] According to an exemplary embodiment, the transformation includes: performing an N1-point 1-D transformation on each of the N1x1 vectors along the x-axis of the 3-D cube, performing an N2-point 1-D transformation on each of the N2x1 vectors along the y-axis of the 3-D cube, and performing a 2-point transformation on each of the 2x1 vectors along the z-axis of the 3-D cube.

[0016] According to an exemplary embodiment, the transformation includes: performing a 2-point 1-D transformation on each of the 2x1 vectors along the z-axis of the 3-D cube, and performing an N1xN2 transformation on the xy plane of the 3-D cube.

[0017] According to an exemplary embodiment, performing a transformation includes performing a primary transformation and a secondary transformation.

[0018] According to an exemplary embodiment, at least one of the primary transformation and the secondary transformation includes setting the coefficient to zero in response to a predetermined condition that determines that the value of the coordinate associated with the coefficient of at least one of the chromaticity prediction residual signal blocks is greater than or equal to a threshold. Attached Figure Description

[0019] These and other objects, features, and advantages will become apparent from the following detailed description of illustrative embodiments, which is taken in conjunction with the accompanying drawings. The various features in the drawings are not drawn to scale because they are illustrated to provide clarity for those skilled in the art to understand in conjunction with the detailed description. In the drawings:

[0020] Figure 1 A network computer environment according to at least one embodiment is shown.

[0021] Figure 2 It is a diagram of the nominal angle of AV1 according to at least one embodiment.

[0022] Figure 3 An operational flowchart illustrating the steps performed by a program encoding video data according to at least one embodiment is shown.

[0023] Figure 4 According to at least one embodiment Figure 1 The diagram depicts the internal and external components of a computer and server.

[0024] Figure 5 It includes, according to at least one embodiment Figure 1 The diagram depicts an illustrative cloud computing environment for a computer system.

[0025] Figure 6 According to at least one embodiment Figure 5 A block diagram illustrating the functional layers of an illustrative cloud computing environment.

[0026] Figure 7 This is a block diagram illustrating the features of a non-directional smooth intra-frame predictor in AVI, according to an exemplary embodiment.

[0027] Figure 8 This is a block diagram of the features of a recursive intra-frame filtering mode according to an exemplary embodiment.

[0028] Figure 9 This is a block diagram illustrating the features of a multi-layer reference frame structure according to an exemplary embodiment.

[0029] Figure 10 This is a block diagram of features constructed from a list of candidate motion vectors according to an exemplary embodiment.

[0030] Figure 11 This is a line graph showing the characteristics of motion field estimation according to an exemplary embodiment.

[0031] Figure 12 This is a block diagram illustrating the characteristics of an overlapping region for overlapped block motion compensation (OMBC) according to an exemplary embodiment.

[0032] Figure 13 This is a block diagram illustrating features related to torsional motion compensation according to an exemplary embodiment.

[0033] Figure 14 This is a block diagram of features related to advanced composite prediction according to an exemplary embodiment.

[0034] Figure 15 This is a flowchart illustrating the features according to an exemplary embodiment.

[0035] Figure 16 It is a spatial diagram of the features of the residuals placed together as a 3D cube, according to an exemplary embodiment.

[0036] Figure 17 This is a block diagram of the features of the system according to an exemplary embodiment.

[0037] Figure 18 This is a block diagram of the features of the decoder according to an exemplary embodiment.

[0038] Figure 19 This is a block diagram of the features of the encoder according to an exemplary embodiment. Detailed Implementation

[0039] This document discloses detailed embodiments of the claimed structures and methods; however, it is to be understood that the disclosed embodiments are merely illustrative of the claimed structures and methods, which can be implemented in various forms. These structures and methods may be embodied in many different forms and should not be construed as limited to the exemplary embodiments described herein. Rather, these exemplary embodiments are provided so that this disclosure is thorough and complete, and fully conveys the scope to those skilled in the art. Details of well-known features and techniques may be omitted in the description to avoid unnecessarily obscuring the presented embodiments.

[0040] The embodiments generally relate to the field of data processing, and more specifically, to video encoding and decoding. Furthermore, the exemplary embodiments described below provide a system, method, and computer program for encoding and / or decoding video data using incremental (delta) angle values ​​derived from nominal angle values. Therefore, some embodiments offer improved computational capabilities by not requiring each incremental angle to be signaled and allowing on-the-fly calculation of the incremental angle values.

[0041] As previously mentioned, AOMedia Video 1 (AV1) is an open video coding format designed for transmitting video over the Internet. It was developed by the Open Media Consortium (AOMedia) as a successor to VP9. AOMedia is a consortium founded in 2015 that includes semiconductor companies, video-on-demand providers, video content producers, software development companies, and web browser vendors. In AV1, there are a total of 56 directional angles, of which 8 are nominal angles and the rest are specified as increments of the nominal angles. However, regardless of the collocated luma prediction mode, the nominal and increment angles in all directional modes are signaled for the chroma component. Furthermore, increment angles are allowed for both luma and chroma intra-frame prediction modes, but the correlation of the increment angles between the luma and chroma components is not used. Therefore, it may be advantageous to derive the increment angle values ​​of the chroma components based on the nominal angles from the luma component, rather than signaling all 56 angle values.

[0042] This document describes various aspects of methods, apparatus (systems), and computer-readable media based on flowchart illustrations and / or block diagrams of various embodiments. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0043] Now refer to Figure 1 A functional block diagram of a network computer environment illustrates a video coding system 100 (hereinafter referred to as the "System") that encodes and / or decodes video data using incremental angles derived from nominal angles. It should be understood that...Figure 1 This is merely a description of one implementation method and does not imply any limitation on the environments in which different embodiments can be implemented. Numerous modifications can be made to the described environment to suit design and implementation requirements.

[0044] System 100 may include computer 102 and server computer 114. Computer 102 may communicate with server computer 114 via communication network 110 (hereinafter referred to as the "network"). Computer 102 may include processor 104 and software program 108, which is stored on data storage device 106 and is capable of interacting with a user and communicating with server computer 114. (Referring to the following...) Figure 4 The discussion suggests that computer 102 may include internal component 800A and external component 900A, and server computer 114 may include internal component 800B and external component 900B. Computer 102 may be, for example, a mobile device, telephone, personal digital assistant, netbook, laptop computer, tablet computer, desktop computer, or any type of computing device capable of running programs, accessing networks, and accessing databases.

[0045] Server computer 114 can also run in a cloud computing service model, which includes, for example, Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS), as described below. Figure 6 and Figure 7 The discussion continues. Server computer 114 can also be located in a cloud computing deployment model, which includes, for example, private cloud, community cloud, public cloud, or hybrid cloud.

[0046] A server computer 114, which can be used to encode video data, allows the execution of a video encoding program 116 (hereinafter referred to as the "program") that can interact with the database 112. See below for further details. Figure 3 The video encoding program method is described in more detail below. In one embodiment, computer 102 may operate as an input device including a user interface, while program 116 may run primarily on server computer 114. In an alternative embodiment, program 116 may run primarily on one or more computers 102, while server computer 114 may be used to process and store data used by program 116. It should be noted that program 116 may be a standalone program or may be integrated into a larger video encoding program.

[0047] However, it should be noted that in some instances, processing for program 116 can be distributed in any proportion between computer 102 and server computer 114. In another embodiment, program 116 can run on more than one computer, server computer, or some combination of computers and server computers, for example, on multiple computers 102 communicating with a single server computer 114 via network 110. In another embodiment, for example, program 116 can run on multiple server computers 114, which communicate with multiple client computers via network 110. Optionally, the program can run on a network server, which communicates with servers and multiple client computers via a network.

[0048] Network 110 may include wired connections, wireless connections, fiber optic connections, or a combination thereof. Typically, network 110 may be any combination of connections and protocols supporting communication between computer 102 and server computer 114. Network 110 may include various types of networks, such as local area networks (LANs), wide area networks (WANs) such as the Internet, telecommunications networks such as the Public Switched Telephone Network (PSTN), wireless networks, public switched networks, satellite networks, cellular networks (e.g., fifth-generation (5G), long-term evolution (LTE), third-generation (3G), code division multiple access (CDMA), etc.), public land mobile networks (PLMNs), metropolitan area networks (MANs), private networks, self-organizing networks, intranets, fiber optic-based networks, etc., and / or combinations of these or other types of networks.

[0049] Figure 1 The number and layout of devices and networks shown are provided as an example. In reality, compared to... Figure 1 The devices and / or networks shown may exist in greater numbers, fewer numbers, different types, or different arrangements. Furthermore, Figure 1 The two or more devices shown can be implemented within a single device, or Figure 1The single device shown can be implemented as multiple distributed devices. Additionally or alternatively, a group of devices in system 100 (e.g., one or more devices) can perform one or more functions described as being performed by another group of devices in system 100.

[0050] Now refer to Figure 2 The diagram 200 illustrates the nominal angles of AV1. VP9 supports eight orientation modes, corresponding to angles from 45 degrees to 207 degrees. To utilize more spatial redundancy in orientation textures, the intra-frame orientation modes in AV1 are extended to have a finer-grained set of angles. In AV1, there are eight nominal angles between 45 and 207 degrees, including V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED. For each nominal angle, there can be seven more refined angles, resulting in a total of 56 orientation angles for AV1. Predicted angles can be represented by adding an angle increment to the nominal interior angle, i.e., -3 to 3 times the 3-degree step. The increment angle may not need to be signaled by the chroma component but can be derived from the corresponding intra-frame prediction mode of the luma.

[0051] In AV1, eight nominal modes and five non-angular smooth modes are first signaled. Then, if the current mode is an angular mode, an index is further signaled to indicate the angular increment relative to the corresponding nominal angle. To implement orientation prediction modes in AV1 in a generic manner, all 56 intra-frame orientation prediction modes in AV1 are implemented using a unified orientation predictor. This unified orientation predictor projects each pixel to a reference sub-pixel position and interpolates the reference pixel using a 2-tap bilinear filter.

[0052] In one or more embodiments, if the current chroma intra-prediction mode is a directional intra-prediction mode and the nominal angle of the chroma intra-prediction mode is equal to the nominal angle of the corresponding luma intra-prediction mode, the increment angle of the chroma component can be set to be equal to the increment angle of the luma intra-prediction mode. Otherwise, the increment angle of the chroma component can be set to 0.

[0053] In one or more embodiments, the increment angle of the chromaticity component can be set to be equal to the increment angle of the luminance component, regardless of whether the nominal modes between luminance and chromaticity are the same.

[0054] In one or more embodiments, if the current chroma intra-prediction mode is a directional intra-prediction mode and the nominal angle of the chroma intra-prediction mode is equal to the nominal angle of the corresponding luma intra-prediction mode, the increment angle of the chroma component can be set to be equal to the increment angle of the luma intra-prediction mode. Otherwise, when the nominal angle of the chroma intra-prediction mode is equal to the left / top adjacent mode of the corresponding luma block, the increment angle of the chroma component can be set to be equal to the increment angle of the left / top adjacent mode of the corresponding luma block. Otherwise, the increment angle of the chroma component can be set to 0.

[0055] In one or more embodiments, when semi-decoupled partitioning is applied, multiple luma blocks may exist associated with a chroma block. Therefore, multiple sample locations can be predefined, and incremental and nominal angles associated with these locations for predicting co-located luma blocks can be identified. One or more of these identified nominal and incremental angles can be used to derive the incremental angle of the current chroma block. In one example, the predefined sample locations may include a top-left sample and / or a center / middle sample. In another example, the predefined sample locations may include four corner samples and a center / middle sample. In yet another example, the incremental angle of the chroma component can be derived using the incremental angle that is most likely to be used in the identified prediction pattern. In yet another example, the predefined sample locations may include two selected locations from four corner samples and one center / middle sample. In yet another example, the predefined sample locations may include three selected locations from four corner samples and one center / middle sample.

[0056] In one or more embodiments, if the current chroma intra-prediction mode is a directional intra-prediction mode and the nominal angle of the chroma intra-prediction mode is equal to the nominal angle of the corresponding luma intra-prediction mode, the increment angle of the chroma component can be set to be equal to the increment angle of the luma intra-prediction mode. Otherwise, when the nominal angle of the chroma intra-prediction mode is equal to the left / top adjacent mode of the current chroma block, the increment angle of the chroma component can be set to be equal to the increment angle of the left / top adjacent mode of the current chroma block. Otherwise, the increment angle of the chroma component can be set to 0.

[0057] The increment angle of the corresponding luma block can be used for entropy coding of the increment angle of the chroma intra-prediction mode. In one or more embodiments, the increment angle of co-located luma blocks can be used as context for entropy coding of the increment angle of the chroma component. In one or more embodiments, the increment angle of adjacent chroma blocks is used as context for entropy coding of the increment angle of the chroma component. In one or more embodiments, the absolute difference between the increment angle of the chroma block and the increment angle of the corresponding luma block can be signaled for entropy coding of the chroma intra-prediction mode, instead of signaling the increment angle of the current chroma block. In one or more embodiments, the above embodiments are applied only when the nominal modes between luma and chroma can be the same or when the absolute difference of the prediction angles between the two modes is within a given threshold.

[0058] According to one or more embodiments, for entropy coding of chroma intra-frame prediction modes, a first flag can be signaled to indicate whether the current mode is chroma-from-luma (CfL). If the first flag is signaled to indicate that a CfL value is not used, a second flag can be signaled to indicate whether the current mode is equal to the nominal mode of the corresponding luma block. If the current mode is equal to the nominal mode of the corresponding luma block, the current mode is an orientation mode, and increment angles are allowed, a third flag can be signaled to indicate the index of the increment angle. Otherwise, a third flag is signaled to indicate which of the remaining nominal modes the current mode is. If the first flag is signaled to indicate that a CfL value is not used, the parameters of the CfL mode can be further signaled.

[0059] According to one or more embodiments, for entropy coding of chroma intra-frame prediction modes, a first flag can be signaled to indicate whether the current mode is equal to the nominal mode or CfL mode of the corresponding luma block. If the first flag is signaled to indicate that the current mode is equal to the nominal mode or CfL mode of the corresponding luma block, a second flag can be signaled to indicate which of the two modes can be the current mode. If the current mode is equal to the nominal mode of the corresponding luma block, the current mode is an orientation mode, and increment angles are allowed, the index of the increment angle can be further signaled. If the current mode is a CfL mode, the parameters of the CfL mode can be further signaled. If the first flag is signaled to indicate that the current mode is not equal to the nominal mode or CfL mode of the corresponding luma block, a second flag can be signaled to indicate which of the remaining nominal modes can be applied to the current chroma block.

[0060] In one or more embodiments, in response to the nominal angles of the chroma component and the luminance component being the same or close to each other, the first incremental angle value can be entropy-coded based on the incremental angles of co-located luminance blocks as context. The nominal angles can be close to each other when the difference between them is less than or equal to 2 degrees. The incremental angle value can be entropy-coded based on the incremental angles of co-located luminance blocks or based on the incremental angles of adjacent chroma blocks as context.

[0061] It is understood that, in one or more embodiments, the incremental angles of the luma block and the chroma block may use separate contexts for entropy coding, rather than sharing the same context in the incremental angles of the luma block and the chroma block.

[0062] Now refer to Figure 3 This describes an operational flowchart illustrating the steps of a method 300 for encoding and / or decoding video data. In some embodiments, Figure 3 One or more process blocks can be generated by computer 102 ( Figure 1 ) and server computer 114 ( Figure 1 ) Execution. In some implementations, Figure 3 One or more process blocks may be executed by another device or group of devices that are separate from or include computer 102 and server computer 114.

[0063] In 302, method 300 includes receiving video data including: (1) a chromaticity component having a first nominal angle and a first increment angle; and (2) a luminance component having a second nominal angle and a second increment angle. The first increment angle depends on the second increment angle.

[0064] In 304, method 300 includes signaling a first increment angle value for the first increment angle based at least on an intra-frame prediction mode associated with the luminance component.

[0065] In 306, method 300 includes encoding and / or decoding video data based on a first increment angle value corresponding to a first increment angle.

[0066] Understandable. Figure 3 This is merely a description of one implementation method and does not imply any limitation on how different embodiments can be implemented. Numerous modifications can be made to the described environment to suit design and implementation requirements.

[0067] Figure 4 According to exemplary embodiments, in Figure 1 The block diagram 400 depicts the internal and external components of the computer. It should be understood that... Figure 4This is merely a description of one implementation method and does not imply any limitation on the environments in which different embodiments may be implemented. Numerous modifications can be made to the described environment to suit design and implementation requirements.

[0068] Computer 102 ( Figure 1 ) and server computer 114 ( Figure 1 ) can include Figure 4 The corresponding sets of internal components 800A, 800B and external components 900A, 900B are shown. Each set of internal components 800 includes one or more processors 820, one or more computer-readable RAMs 822 and one or more computer-readable ROMs 824, one or more operating systems 828, and one or more computer-readable tangible storage devices 830, all located on one or more buses 826.

[0069] Processor 820 is implemented in hardware, firmware, or a combination of hardware and software. Processor 820 is a central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), microprocessor, microcontroller, digital signal processor (DSP), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or other type of processing component. In some embodiments, processor 820 includes one or more processors that can be programmed to perform functions. Bus 826 includes components that allow communication between internal components 800A, 800B.

[0070] One or more operating systems 828, software programs 108 ( Figure 1 ) and located on server computer 114 ( Figure 1 The video encoding program 116 on ) Figure 1 The data is stored on one or more corresponding computer-readable tangible storage devices 830 for execution by one or more corresponding processors 820 via one or more corresponding RAMs 822 (typically including cache memory). Figure 4In the illustrated embodiment, each computer-readable tangible storage device 830 is a disk storage device of an internal hard disk drive. Alternatively, each computer-readable tangible storage device 830 is a semiconductor storage device, such as a ROM 824, EPROM, flash memory, optical disc, magneto-optical disc, solid-state drive, compact disc (CD), digital versatile disc (DVD), floppy disk, cassette tape, magnetic tape, and / or other types of non-transitory computer-readable tangible storage devices capable of storing computer programs and digital information.

[0071] Each set of internal components 800A, 800B also includes an R / W drive or interface 832 for reading from or writing to one or more portable computer-readable physical storage devices 936 (e.g., CD-ROM, DVD, memory stick, magnetic tape, disk, optical disc, or semiconductor storage device). For example, software program 108 ( Figure 1 ) and video encoding program 116 ( Figure 1 The software program can be stored on one or more corresponding portable computer-readable tangible storage devices 936, and read and loaded into the corresponding hard disk drive 830 via the corresponding R / W drive or interface 832.

[0072] Each set of internal components 800A, 800B also includes a network adapter or interface 836 (e.g., a TCP / IP adapter card), a wireless Wi-Fi interface card, or a 3G, 4G, 5G wireless interface card, or other wired or wireless communication links. Software program 108 ( Figure 1 ) and server computer 114 ( Figure 1 The video encoding program 116 on ) Figure 1 The computer can download data from an external computer to the computer 102 via a network (e.g., the Internet, a local area network, or another wide area network) and a corresponding network adapter or interface 836. Figure 1 The network includes a network adapter or interface 836 and a server computer 114. From the network adapter or interface 836, software program 108 and video encoding program 116 on the server computer 114 are loaded into the corresponding hard disk drive 830. The network may include copper wire, fiber optic, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers.

[0073] Each set of external components 900A, 900B may include a computer monitor 920, a keyboard 930, and a computer mouse 934. External components 900A, 900B may also include a touchscreen, virtual keyboard, touchpad, pointing device, and other human-machine interface devices. Each set of internal components 800A, 800B also includes a device driver 840 that interfaces with the computer monitor 920, keyboard 930, and computer mouse 934. Device driver 840, R / W driver or interface 832, and network adapter or interface 836 include hardware and software (stored in storage device 830 and / or ROM 824).

[0074] It should be understood in advance that although this disclosure includes a detailed description of cloud computing, the implementation of the teachings described herein is not limited to a cloud computing environment. Rather, some embodiments can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0075] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing power, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with service providers. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0076] The characteristics are as follows:

[0077] On-demand self-service: Cloud consumers can automatically and unilaterally provide computing power (such as server time and network storage) as needed without requiring manual interaction with the service provider.

[0078] Extensive network access: Capabilities can be obtained through the network and accessed through standard mechanisms that facilitate use by heterogeneous thin client platforms or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0079] Resource pooling: Pooling a provider's computing resources to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. There is a sense of location independence because consumers typically cannot control or know the exact location of the resources provided, but can specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0080] Rapid elasticity: Capacity can be provided quickly and flexibly (in some cases automatically) to rapidly expand outward and rapidly contract inward. For consumers, the capacity available for supply often appears unlimited and can be purchased at any time and in any quantity.

[0081] Measuring services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be detected, controlled, and reported, providing transparency to both service providers and consumers.

[0082] The service model is as follows:

[0083] Software as a Service (SaaS): The capability offered to consumers is the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from various client devices via a thin client interface, such as a web browser (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even the individual application capabilities, with possible exceptions of limited user-specific application configuration settings.

[0084] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created or acquired by the consumer onto cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage; instead, they control the deployed applications and the configuration of any application hosting environments.

[0085] Infrastructure as a Service (IaaS): This provides consumers with the capability to deliver processing, storage, networking, and other basic computing resources that enable them to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure; instead, they control the operating system, storage, deployed applications, and may have limited control over the selection of networking components (e.g., host firewalls).

[0086] The deployment model is as follows:

[0087] Private cloud: Cloud infrastructure designed specifically for an organization's operation. The cloud infrastructure can be managed by the organization or a third party and can exist in an on-premises or off-premises manner.

[0088] Community cloud: A cloud infrastructure shared by several organizations that supports a specific community with common concerns (e.g., missions, security requirements, policies, and compliance considerations). A community cloud can be managed by these organizations or a third party and can exist in an on-premises or off-premises manner.

[0089] Public cloud: Cloud infrastructure available to the general public or large industrial groups and owned by organizations that sell cloud services.

[0090] Hybrid cloud: Cloud infrastructure is a combination of two or more clouds (private, community, or public) that maintain distinct entities but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursts for load balancing between clouds).

[0091] Cloud computing environments are service-oriented, emphasizing statelessness, loose coupling, modularity, and semantic interoperability. At its core is the infrastructure comprising a network of interconnected nodes.

[0092] Reference Figure 5 This document describes an exemplary cloud computing environment 500. As shown, the cloud computing environment 500 includes one or more cloud computing nodes 10, and local computing devices used by cloud consumers (e.g., personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N) can communicate with the cloud computing nodes 10. The cloud computing nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks (e.g., private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above). This allows the cloud computing environment 500 to provide infrastructure, platforms, and / or software as services that do not require cloud consumers to maintain resources on their local computing devices. It is understood that... Figure 5 The types of computing devices 54A-N shown are for illustrative purposes only, and cloud computing node 10 and cloud computing environment 500 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0093] Reference Figure 6 This demonstrates the 500 (cloud computing environment) Figure 5 The 600 provides a set of functional abstraction layers. This should be understood beforehand. Figure 6 The components, layers, and functions shown are for illustrative purposes only, and the embodiments are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0094] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a mainframe 61; a server 62 based on a RISC (Reduced Instruction Set Computer) architecture; a server 63; a blade server 64; a storage device 65; and a network and network components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0095] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 71; virtual storage 72; virtual network 73, including virtual private network; virtual application and operating system 74; and virtual client 75.

[0096] In one example, management layer 80 can provide the following functionalities: Resource Provisioning 81 provides dynamic acquisition of computing resources and other resources for performing tasks within the cloud computing environment. Metering and Pricing 82 provides cost tracking and billing or invoicing for the consumption of these resources as they are utilized in the cloud computing environment. In one example, these resources may include application software licenses. Security Aspects provide authentication for cloud consumers and tasks, as well as protection for data and other resources. User Access Points 83 provide access to the cloud computing environment for consumers and system administrators. Service Level Management 84 provides allocation and management of cloud computing resources to meet the required service level. Service Level Agreement (SLA) Planning and Implementation 85 provides pre-arrangement and acquisition of cloud computing resources, anticipating future demand for cloud computing resources according to the SLA.

[0097] Workload layer 90 provides examples of functionalities that can be leveraged in a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analytics and processing 94; transaction processing 95; and video encoding / decoding 96. Video encoding / decoding 96 can encode / decode video data using incremental angles derived from the nominal angle.

[0098] Figure 7This is a block diagram 700 showing features of a non-directional smooth intra-predictor in AVI, according to an exemplary embodiment. For example, in AV1, there are five non-directional smooth intra-predictor modes: DC, PAETH, SMOOTH, SMOOTH_V, and SMOOTH_H. For DC prediction, the average of the left and above neighboring samples is used as the predictor for the block to be predicted. For the PAETH predictor, reference samples are first obtained from the top, left, and top-left sides, and then the closest (top + left - top-left) value is set as the predictor for the pixel to be predicted. Figure 7 This shows the positions of the top, left, and upper left samples of a pixel in the current block. For SMOOTH, SMOOTH_V, and SMOOTH_H modes, blocks are predicted using quadratic interpolation in the vertical or horizontal direction or by averaging the two directions.

[0099] Figure 8 This is a block diagram 800 illustrating the features of recursive intra-frame filtering modes according to an exemplary embodiment. For example, to capture the attenuation spatial correlation of edge references, a FILTER INTRA mode for luma patches is designed. Five filter intra-frame modes are predefined for AV1, each represented by a set of eight 7-tap filters to reflect the correlation between a pixel in a 4x2 patch and its seven neighboring pixels. In other words, the weight factor of the 7-tap filter is position-dependent. For example, an 8x8 patch is divided into eight 4x2 patches, such as... Figure 8 As shown. These small pieces are in Figure 8 The sub-blocks are represented by B0, B1, B2, B3, B4, B5, B6, and B7. For each sub-block, its seven neighboring blocks (represented by R0 to R6) are used to predict the pixels in the current sub-block. For sub-block B0, all neighboring blocks are reconstructed. However, for other sub-blocks, some neighboring blocks are not reconstructed, and the predicted values ​​of the directly adjacent blocks are used as a reference. For example, none of the neighboring blocks of sub-block B7 are reconstructed, so according to an embodiment, the predicted samples of the neighboring blocks are used instead.

[0100] CfL is an intra-frame predictor used only for chroma, which models chroma pixels as a linear function of the reconstructed luminance pixels. The CfL prediction is expressed as follows (Equation 1):

[0101] CfL(α)=α×L AC +DC Formula 1

[0102] Among them, L ACThe AC contribution of the luminance component is represented by α, the parameter of the linear model is represented by α, and the DC contribution of the chrominance component is represented by DC. Specifically, the reconstructed luminance pixels are second-sampled to chrominance resolution, and then the average value is subtracted to form the AC contribution. To approximate the chrominance AC component with the AC contribution, instead of requiring the decoder to calculate scaling parameters as in some existing technologies, AV1 CfL determines the parameter α based on the original chrominance pixels and signals the parameter α in the bitstream. This reduces the complexity of the decoder and produces more accurate predictions. As for the DC contribution of the chrominance component, it is calculated using an intra-frame DC model, which is sufficient for most chrominance content and is mature and can be implemented quickly.

[0103] Figure 9 This is a block diagram 900 showing features of a multi-layer reference frame structure according to an exemplary embodiment. For example, there may be extended reference frames, such that in addition to the three reference frames of VP9, ​​namely LAST (most recent past) frame, GOLDEN (distant past) frame, and ALTREF (temporally filtered future) frame, there are four other types of reference frames:

[0104] LAST2 and LAST3: Two nearby past frames

[0105] • BWDREF, ALTREF2: Two future frames

[0106] • BWDREF is a leading frame that is directly encoded without temporal filtering, making it more suitable as a backward reference for relatively short distances.

[0107] ALTREF2 is a future reference for intermediate filtering between GOLDEN and ALTREF.

[0108] Figure 10 This is a block diagram 1000 of features for constructing a list of candidate motion vectors according to an exemplary embodiment. For example, AV1 combines a motion vector reference selection scheme to efficiently encode motion vectors through dynamic spatial and temporal motion vector references. AV1 uses a wider spatial neighborhood than its predecessor VP9, ​​as well as a motion field estimation process, to find temporal motion vector reference candidates.

[0109] Figure 11 This is a line graph 1100 showing the characteristics of motion field estimation according to an exemplary embodiment. For example, the motion field estimation process searches for all motion trajectories through an 8x8 block resolution in a juxtaposed 128x128 region. For example, in Figure 11 In the process described above, the motion vector MV is searched. Ref2It then derives projected motion vectors, such as MV0 and MV1. Once all candidate motion vectors are found, they are sorted, merged, and ranked to establish up to four final candidates. AV1 then signals the index of the selected reference motion vector from the list and optionally encodes the motion vector difference.

[0110] Figure 12 This is a block diagram 1200 illustrating features of overlapping regions for overlapped block motion compensation (OMBC) according to an exemplary embodiment. For example, to reduce prediction errors around block boundaries by combining predictions obtained from adjacent motion vectors, AV1 progressively combines block-based predictions with secondary predictors from the top and left edges by applying 1-D filters in the vertical and horizontal directions, respectively. As an example, in Figure 12 In this process, the shadow region of prediction block 0 will be predicted by recursively generating mixed prediction samples through a 1-D raised cosine filter. Figure 12 The shadowed areas are shown using the top neighbor 2 and the left neighbor 4, respectively.

[0111] Figure 13 This is a block diagram 1300 illustrating features related to warp motion compensation according to an exemplary embodiment. For example, AV1 proposes two affine prediction models: global warp motion compensation and local warp motion compensation. The former signals the frame-level affine model between the frame and its reference, while the latter implicitly handles varying local motion with minimal overhead. Local motion parameters are derived at the block level using 2D motion vectors from the causal neighborhood. This affine model is implemented through successive horizontal and vertical shearing operations using an 8-tap interpolation filter based on 1 / 64 pixel precision. Figure 13 The two-step twisting process, first horizontal shearing and then vertical shearing, is shown.

[0112] Figure 14 This is a block diagram 1400 of features relating to advanced composite prediction according to an exemplary embodiment. For example, according to an embodiment, AV1 may allow more versatile predictions at each pixel location (i,j), as shown below (Definition 1).

[0113]

[0114] Here, p1 and p2 are two predictions, and m(i,j) is the weighting factor located in [0,1] from the predefined table.

[0115] For composite wedge prediction, a predefined set of 16 possible wedge partitions is provided in AV1, and the selected wedge index is signaled. For square and rectangular blocks, the orientation includes horizontal, vertical, and tilted (with a tilt of ±2 or ±0.5). To mitigate artifacts, a two-dimensional wedge mask with a soft cliff shape is employed.

[0116] For differential modulation mask prediction, according to the embodiment, AV1 can allow some regions of the prediction to come more from one prediction than from another. More specifically, a mask of the following form (“Definition 2”) is used:

[0117]

[0118] Here, b controls the influence of the first predictor, and a ensures smooth modulation.

[0119] For frame distance-based composite prediction, according to an embodiment, AV1 may include a modified weighting scheme that takes into account frame distance, which is defined as the absolute difference between the timestamps of two frames. In order to reduce quantization noise by considering not only temporal distance but also the averaging effect through multiple references, AV1 adopts a frame distance-based scheme while giving slightly more weight to the far-distance predictor, as shown below ("Definition 3" and "Definition 4").

[0120]

[0121]

[0122] Here, p is the prediction based on frame distance, and p1 and p2 are the prediction values ​​from two reference frames. The weights (w1, w2) are determined above, where d1 and d2 represent the frame distances between the current frame and the two reference frames.

[0123] For composite inter-frame-intra-frame prediction, according to the embodiments, AV1 can combine intra-frame prediction and inter-frame prediction, and supports four commonly used intra-frame modes for the intra-frame portion. There are two types of masks associated with this mode: (i) a smooth wedge mask similar to the mask used for inter-frame-intra-frame modes; and (ii) a mode-dependent mask that decays along the main direction of the intra-frame mode.

[0124] According to an exemplary embodiment, for each transform unit, AV1 coefficient encoding begins with a signal indicating a skip sign, followed by a signal indicating the transform kernel type and the end-of-block (eob) position when the skip sign is zero. Each coefficient value is then mapped to multiple level maps and symbols.

[0125] After encoding the eob position, the lower-level map and the middle-level map are encoded in reverse scan order. The lower-level map indicates whether the coefficient size is between 0 and 2, and the middle-level map indicates whether the range is between 3 and 14. Next, the signs of the coefficients are encoded in forward scan order, and residuals of coefficients greater than 14 are encoded using Exp-Golomb codes.

[0126] Regarding the use of context modeling, low-level graph coding includes the size and direction of the transform, as well as information on up to five neighboring coefficients. On the other hand, mid-level graph coding follows a similar approach to low-level graph coding, except that the number of neighboring coefficients is reduced to two. The symbols used for Exp-Golomb codes at the residual level and for AC coefficients do not require encoding using any context model, while the symbols for DC coefficients are encoded using the DC symbols of their neighboring transform units.

[0127] VVC Draft 6 supports modes for joint encoding of chroma residuals. The use (activation) of a joint chroma encoding mode is indicated by the TU-level flag `tu_joint_cbcr_residual_flag`, and the selected mode is implicitly indicated by the chroma-coded block flags (CBFs). The flag `tu_joint_cbcr_residual_flag` is present if one or both of the TU's chroma CBFs are equal to 1. In the PPS and slice headers, the chroma quantization parameter QP offset values ​​are signaled for the joint chroma residual encoding mode to distinguish them from the chroma QP offset values ​​signaled for the regular chroma residual encoding mode. These chroma QP offset values ​​are used to derive chroma QP values ​​for those blocks encoded using the joint chroma residual encoding mode. When the corresponding joint chroma encoding mode (Mode 2 in Table 1) is activated in a TU, the chroma QP offset is added to the applied luma-derived chroma QP during quantization and decoding of that TU. For other modes (Mode 1 and Mode 3 in Table 1), the chroma QP is derived in the same way as the regular Cb or Cr blocks. Table 1 depicts the reconstruction process of the chroma residuals (resCb and resCr) of the transport transform block. When this mode is activated, a single joint chroma residual block (resJointC[x][y] in Table 1) is signaled, and information such as tu_cbf_cb, tu_cbf_cr, and Csign (CSign is the sign value specified in the slice header) is taken into account to derive the residual block of Cb (resCb) and the residual block of Cr (resCr).

[0128] The three joint chroma coding modes mentioned above are supported only in intra-frame coded CUs. In inter-frame coded CUs, only mode 2 is supported. Therefore, for inter-frame coded CUs, the syntax element tu_joint_cbcr_residual_flag exists only when both chroma CBFS values ​​are 1. Table 1 below indicates the reconstruction of the chroma residuals, where the value CSign is the sign value (+1 or -1) specified in the slice header, and resJointC[][] is the transmitted residual.

[0129] Table 1

[0130]

[0131] Figure 15 This is a flowchart 1500 showing the features according to an exemplary embodiment.

[0132] Figure 16 It is a spatial diagram of the features of the residuals placed together as a 3D cube, according to an exemplary embodiment. Figure 16 The features described in relation to the other figures herein can be used individually or in any combination in any order. Furthermore, each method (or embodiment), encoder, and decoder can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium. Assume one or more N1xN2 Cb prediction residual blocks and their corresponding Cr prediction residual blocks are used as inputs at S151.

[0133] At S151, the two N1xN2 Cb and Cr prediction residual signal blocks are transformed into one or more other signal blocks of size MxL, where MxL is less than or equal to 2xN1xN2; however, one or more modifications can be replaced or added at S151.

[0134] For example, improvements to the technical problems of compression described above can be achieved through the transformations described in this paper, which remove redundancy between the chroma prediction residual and any two of similar signals.

[0135] For example, according to an exemplary embodiment, the N1xN2 Cb residual block and its corresponding Cr residual block are grouped together as a single block, and a transformation is performed on this block. A 2D transformation can exist where each input channel is vectorized (also called vectorization) separately through various scan sequences to achieve transform coefficients of size LxM (referred to herein as "Type I"). A 2D transformation can also exist where, in addition to the scan sequences mentioned above, the two input channels are vectorized through interleaving (referred to herein as "Type II").

[0136] As a more specific embodiment of Type I and Type II, according to the exemplary embodiment, the two N1xN2 transform coefficient blocks can be encoded separately for one or more of the following:

[0137] -(2xN1xN2) multiplied by (2xN1xN2)KLT (Karhunen-Loève transform),

[0138] - Dimensional Reduction KLT (RD-KLT)

[0139] -RD-KLT's separable approximation

[0140] -Givens / permutation matrix decomposition, and

[0141] - Combinations of trigonometric transformations.

[0142] In one embodiment, when Cb and Cr are grouped together as a block at S151, the Cb and Cr residuals are vectorized separately according to a sawtooth, raster, or diagonal scan order, and then concatenated to form a (2xN1xN2) x 1 vector. This vector is transformed by a (LxM) x (2xN1xN2) matrix to form a signal block of size (LxM) x 1. The coefficients of this signal block are quantized, and the quantization index is signaled. The decoding process dequantizes the index and inversely transforms it by a (2xN1xN2) x (LxM) matrix to obtain a reconstructed 2xN1xN2 vector, which can be scanned back to the reconstructed Cb and Cr residual block.

[0143] In one embodiment, when Cb and Cr are grouped together as a block at S151, the Cb residual block and its corresponding Cr residual block are vectorized in an interleaved manner (i.e., one sample comes from Cb, the next sample comes from Cr), where each sample is obtained from the residual block in a zigzag, raster, or diagonal scan order to form a (2xN1xN2) x 1 vector. This vector is then transformed by a (LxM) x (2xN1xN2) matrix to form a transformed signal block of size (LxM) x 1. The coefficients of this transformed signal block are quantized, and the quantization index is signaled. The decoding process dequantizes the index and inversely transforms it by a (2xN1xN2) x (LxM) matrix to obtain a reconstructed 2xN1xN2 vector, which can be scanned back to the reconstructed Cb and Cr residual blocks.

[0144] In one embodiment, at S151, there is also a setting where LxM equals 2xN1xN2, and at S153, the output of the transform is further divided into two N1xN2 blocks, which are further quantized and decoded or entropy encoded into the bitstream.

[0145] In one embodiment, at S151, when LxM equals 2xN1xN2, the size of the above transformation matrix is ​​(2xN1xN2) multiplied by (2xN1xN2), and can be determined as KLT (Karhunen-Loève transform). Given a 2xN1xN2 multiplied by 1 input vector signal X (assuming it is mean-removed to have zero mean), then the (2xN1xN2) multiplied by (2xN1xN2) covariance matrix... It satisfies the following formula (“Formula 2”).

[0146] R XX I j =λ j I j Formula 2

[0147] Where, λ j and I j It is R XX The j-th eigenvalue and its corresponding eigenvector.

[0148] The KLT matrix H uses the aforementioned eigenvectors as rows of the matrix and transforms X into Y, as shown below (“Equation 3”).

[0149] Y=HX Formula 3

[0150] in,

[0151] The elements of Y are uncorrelated, and their variance is equal to the eigenvalues. Furthermore, H is an orthogonal transform such that the input signal vector X can be recovered as follows (“Equation 4”).

[0152] X = H T Y formula 4

[0153] In one embodiment, at S151, the above (LxM) multiplied by (2xN1xN2) matrix can be determined by selecting only LxM eigenvectors corresponding to LxM largest eigenvalues ​​from the (2xN1xN2) basis vectors of KLT.

[0154] In one embodiment, at S151, the (LxM) multiplied by (2xN1xN2) KLT determined above can be approximated by its separable approximation. As an example, the KLT matrix H can be approximated by solving the following optimization problem (Prob 1).

[0155] min P,S ‖H-P1SP2‖ F Question 1

[0156] Where S is a separable row / column matrix, and P1 and P2 are permutation matrices to be determined.

[0157] In one embodiment, at S151, the above (LxM) multiplied by (2xN1xN2) matrix can be constructed as a multi-level Givens rotation matrix and / or permutation matrix. As an example, this KLT matrix H can be approximated by solving the following optimization problem (Prob 2).

[0158] min T ‖HT‖ F Question 2

[0159] in, It is a multi-level cascade, with each level consisting of the product of a Givens rotation matrix, a permutation matrix, and its inverse.

[0160] In one embodiment, at S151, the above (LxM) multiplied by (2xN1xN2) matrix can be adaptively constructed according to a set of DCT / DST (Discrete Cosine / Discrete Sine) transforms.

[0161] At S154, it can be determined whether to place the N1xN2 Cb residual block and its corresponding Cr residual block together as a 2xN1xN2 three-dimensional (3-D) cube (e.g. Figure 16 If the cube is 1600, then perform a 3-D transformation on the cube at S155.

[0162] According to exemplary embodiments, either a fully separable 3D transformation or a 1D+2D transformation may exist for use with the features described herein.

[0163] In one embodiment, at S155, the three-dimensional (3-D) transformation can be performed separately along the three axes. That is, an N1-point 1-D transformation is performed on each of the N1x1 vectors along the x-axis, an N2-point 1-D transformation is performed on each of the N2x1 vectors along the y-axis, and another 2-point transformation is performed on each of the 2x1 vectors along the z-axis. The order in which these transformations are applied can be arbitrary.

[0164] In one embodiment, at S155, the three-dimensional (3-D) transformation can be performed separately along the z-axis while performing an arbitrary 2-D transformation on the xy-plane. That is, a 2-point 1-D transformation is performed on each of the 2x1 vectors along the z-axis, while an (N1xN2) transformation is performed on the xy-plane.

[0165] At S156, it can be determined whether to further apply one or more of the transformations described with respect to S152 as a secondary transformation, the input of which is the primary transform coefficient of the chromaticity prediction residual.

[0166] For example, for each N1xN2 chromaticity residual, it takes only K1 transform coefficients, and these two sets of K1 coefficients are concatenated to form the input of the quadratic transform described with respect to S152, and a particular exemplary embodiment may include one or more of the following:

[0167] - Restrictions on the type of primary transformation to which a second transformation will be applied.

[0168] - Some coefficients input to the second transformation are reduced to zero, and

[0169] - Encode only the primary transform coefficients and other coefficients separately.

[0170] If this is the case from S156, then in one embodiment, at S152 (from S156 to S152), the input to the forward secondary transform is a concatenation of two sets of first K1 (K1 <= N1 × N2) coefficients along the forward scan order, each coefficient coming from the transformed Cb and Cr residuals, and the output of the forward secondary transform of size K2 × (2 × K1) is a set of K2 coefficients that replace the same two sets of K1 input coefficients. This set of K2 coefficients is entropy-encoded separately from the two sets of (N1 × N2 - K1) coefficients for which the secondary transform is not applied. The input to the inverse secondary transform of size (2 × K1) × K2 is a quantized coefficient vector of size K2 × 1, and the output of the inverse secondary transform is a coefficient vector of size (2 × K1), whereby each of the K1 coefficients is concatenated with each of the separately entropy-encoded (N1 × N2 - K1) coefficients. In one embodiment, the coefficient is always set to 0 if the coordinates (x, y) associated with the coefficient satisfy the condition that x + y is greater than or equal to a given threshold T. Example values ​​for T include, but are not limited to, integers between 0 and 32. In one embodiment, coefficients with both primary and secondary transformations applied, as well as coefficients with only the primary transformation applied, are encoded separately. That is, coefficients with both primary and secondary transformations applied can be encoded first, followed by coefficients with only the primary transformation applied, and vice versa.

[0171] Although flowchart 1500 has been described with respect to chroma, this document also discloses that flowchart 1500 may include other embodiments. In one embodiment, the method is applied to either the luminance prediction residual or the chroma prediction residual described above. In one embodiment, where applicable, the method is applied between selected two components of an arbitrary multi-component signal. In one embodiment, any of the methods in S152 and S156 through S152 can be applied to the entire M-component signal by cascading M input blocks. In one embodiment, the method regarding S155 can be applied to the entire M-component signal by performing an M-dimensional transformation.

[0172] In other words, this can be extended to other types of multichannel signals, allowing the residual from luminance to be correlated with the residual from chrominance, and generic multichannel signals can also benefit. The 2D transformation method of S152 and the quadratic transformation method of S152 from S156 can be applied by cascading M input blocks to form the input in the case of an M-channel signal, and M-dimensional transformation can be applied.

[0173] Output 153 could be related to what is described in this article and what follows. Figures 17-19 The features discussed further include the output of any one or more of the encoding and decoding processes.

[0174] As an example of the application of the disclosed topic. Figure 17 The diagram illustrates the placement of a video encoder and decoder in a streaming environment. The disclosed principles are equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media such as CDs, DVDs, and memory sticks.

[0175] The streaming system may include an acquisition subsystem 1703, which may include a video source 1701, such as a digital camera, that creates, for example, an uncompressed video sample stream 1713. The sample stream 1713 may be emphasized as having a high data volume compared to the encoded video bitstream and may be processed by an encoder 1702 coupled to the camera 1701. The encoder 1702 may include hardware, software, or a combination of hardware and software to enable or implement aspects of the disclosed subject matter as described in more detail below. The encoded video bitstream 1704 may be emphasized as having a lower data volume compared to the sample stream and may be stored on a streaming server 1705 for future use. One or more streaming clients 1712 and 1707 may access the streaming server 1705 to retrieve copies 1708 and 1706 of the encoded video bitstream 1704. Client 1712 may include video decoder 1711, which decodes an incoming copy 1708 of the encoded video bitstream and creates an output video sample stream 1710 that can be displayed on display 1709 or other display device (not depicted). In some streaming systems, the video bitstream 1704, 1706, and 1708 may be encoded according to certain video encoding / compression standards. Examples of these standards have been mentioned above and are further described herein.

[0176] Figure 18 This can be a functional block diagram of a video decoder 1800 according to an embodiment of the present invention.

[0177] Receiver 1802 may receive one or more encoded video sequences to be decoded by decoder 1800; in the same embodiment or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the decoding of other encoded video sequences. Encoded video sequences may be received from channel 1801, which may be a hardware / software link to a storage device storing the encoded video data. Receiver 1802 may receive encoded video data and other data, such as encoded audio data and / or auxiliary data streams, which may be forwarded to their respective user entities (not depicted). Receiver 1802 may separate the encoded video sequences from other data. To prevent network jitter, buffer memory 1803 may be coupled between receiver 1802 and entropy decoder / resolver 1804 (hereinafter referred to as the "resolver"). Buffer memory 1803 may not be required when receiver 1802 receives data from a store / forward device with sufficient bandwidth and controllability or from a synchronization network, or buffer memory 1803 may be made very small. For use on packet networks such as the Internet, a buffer memory 1803 may also be required, which can be relatively large and can advantageously have an adaptive size.

[0178] Video decoder 1800 may include parser 1804 to reconstruct symbols 1813 from an entropy-encoded video sequence. These symbols may include information for managing the operation of decoder 1800, and potential information for controlling a display device (e.g., display 1812), which is not part of the decoder but may be coupled to it. Control information for the display device may be in the form of supplementary enhancement information (SEI) or fragments of video usability information (VUI) parameter sets (not depicted). Parser 1804 may perform parsing / entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Parser 1804 may extract a subset of parameters from the encoded video sequence for at least one subset of pixels in a subgroup for use in the video decoder, based on at least one parameter corresponding to a group. Subgroups can include Groups of Pictures (GOPs), images, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The entropy decoder / parser can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0179] The parser 1804 can perform entropy decoding / parsing operations on the video sequence received from the buffer memory 1803 to create symbols 1813. The parser 1804 can receive encoded data and selectively decode specific symbols 1813. Furthermore, the parser 1804 can determine whether certain symbols 1813 will be provided to the motion compensation prediction unit 1806, the scaler / inverse transform unit 1805, the intra-frame prediction unit 1807, or the loop filter 1811.

[0180] The reconstruction of symbol 1813 can involve multiple different units, depending on the type of encoded video image or its portions (e.g., inter-frame and intra-frame images, inter-frame and intra-frame blocks) and other factors. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the encoded video sequence by parser 1804. For brevity, the flow of such subgroup control information between parser 1804 and the various units described below is not depicted.

[0181] In addition to the functional blocks already mentioned, the decoder 1800 can be conceptually subdivided into several functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and may be at least partially integrated with one another. However, for the purposes of describing the disclosed subject matter, a conceptual subdivision into the functional units described below is appropriate.

[0182] The first unit is the scaler / inverse transform unit 1805. The scaler / inverse transform unit 1805 receives quantization transform coefficients as symbols 1813 from the parser 1804, along with control information including the transform method used, block size, quantization factor, and quantization scaling matrix. The scaler / inverse transform unit 1805 can output a block containing sample values, which can be input into the aggregator 1810.

[0183] In some cases, the output samples of the scaler / inverse transform 1805 may belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed images, but can use predictive information from previously reconstructed portions of the current image. Such predictive information can be provided by the intra-image prediction unit 1807. In some cases, the intra-image prediction unit 1807 uses surrounding reconstructed information extracted from the current (partially reconstructed) image 1809 to generate blocks of the same size and shape as the blocks being reconstructed. In some cases, the aggregator 1810 adds the predictive information generated by the intra-prediction unit 1807 to the output sample information provided by the scaler / inverse transform unit 1805 based on each sample.

[0184] In other cases, the output samples of the scaler / inverse transform unit 1805 may belong to inter-frame coding and potential motion compensation blocks. In this case, the motion compensation prediction unit 1806 can access the reference image buffer 1808 to extract samples for prediction. After motion compensation is performed on the extracted samples according to the symbols 1813 belonging to the block, these samples can be added by the aggregator 1810 to the output of the scaler / inverse transform unit (referred to as residual samples or residual signals in this case) to generate output sample information. The motion compensation prediction unit may obtain the prediction samples from the address within the reference image buffer under motion vector control, and the motion vector is available to the motion compensation unit in the form of symbols 1813, which may have, for example, X, Y, and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory, motion vector prediction mechanisms, etc., when using subsample precise motion vectors.

[0185] The output samples of aggregator 1810 can be subjected to various loop filtering techniques in loop filter unit 1811. Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video bitstream and available to loop filter unit 1811 as symbols 1813 from parser 1804. However, video compression techniques may also respond to metadata obtained during decoding of a previous (in decoding order) portion of an encoded image or encoded video sequence, and to previously reconstructed and loop-filtered sample values.

[0186] The output of the loop filter unit 1811 can be a sample stream, which can be output to the display device 1812 and stored in the reference image buffer 1808 for subsequent inter-frame image prediction.

[0187] Once fully reconstructed, some of the encoded images can be used as reference images for future predictions. Once the encoded images have been fully reconstructed and the encoded images (e.g., by parser 1804) are identified as reference images, the current reference image 1809 can become part of the reference image buffer 1808, and new current image memory can be reallocated before the reconstruction of subsequent encoded images begins.

[0188] The video decoder 1800 can perform decoding operations according to a predetermined video compression technology. An encoded video sequence conforms to the syntax specified by the video compression technology or standard used, in the sense that the encoded video sequence follows the syntax of such a video compression technology document or standard, and the syntax of the video compression technology or standard specified in the configuration file of the video compression technology document or standard. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (measured in megasamples per second, for example), maximum reference image size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and metadata managed by the HRD buffer, which is signaled in the encoded video sequence.

[0189] In this embodiment, receiver 1802 may receive additional (redundant) data along with the encoded video. This additional data may be included as part of the encoded video sequence. The additional data may be used by video decoder 1800 to properly decode the data and / or more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0190] Figure 19 This may be a functional block diagram of a video encoder 1900 according to an embodiment of this application.

[0191] Encoder 1900 can receive video samples from video source 1901 (which is not part of the encoder), and video source 1901 can capture video images that will be encoded by encoder 1900.

[0192] Video source 1901 can provide a source video sequence encoded by encoder (1703) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, video source 1901 can be a storage device storing pre-prepared video. In a video conferencing system, video source 1901 can be a camera that captures local image information as a video sequence. Video data can be provided as multiple individual images, which are given motion when viewed sequentially. The images themselves can be constructed as spatial pixel arrays, where each pixel may include one or more samples depending on the sampling structure, color space, etc., used. Those skilled in the art can easily understand the relationship between pixels and samples. The following focuses on describing samples.

[0193] According to an embodiment, encoder 1900 can encode and compress images of a source video sequence into an encoded video sequence 1910 in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of controller 1902. The controller controls and is functionally coupled to other functional units described below. For simplicity, coupling is not shown in the figures. Parameters set by the controller may include rate control related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art will readily recognize other functions of controller 1902, as these functions relate to a video encoder 1900 optimized for a particular system design.

[0194] Some video encoders operate within an “encoding loop” readily recognizable to those skilled in the art. In a simplified description, the encoding loop may include an encoding portion of encoder 1902 (hereinafter referred to as the “source encoder”) (responsible for creating symbols based on the input image to be encoded and a reference image) and a (local) decoder 1906 embedded in encoder 1900, which reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (since any compression between the symbols and the encoded video bitstream is lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream is input to reference image memory 1905. Since decoding of the symbol stream produces bit-accurate results independent of the decoder location (local or remote), the contents of the reference image buffer are also bit-accurate between the local and remote encoders. In other words, the reference image samples “seen” by the encoder’s prediction portion are exactly the same as the sample values ​​“seen” by the decoder during decoding when using prediction. The fundamental principle of this reference image synchronization (and the drift that occurs, for example, when synchronization cannot be maintained due to channel errors) is well known to those skilled in the art.

[0195] The operation of the "local" decoder 1906 can be combined with the above. Figure 18 The operation is the same as the "remote" decoder 1800 described in detail. However, please refer to another brief reference. Figure 19 When symbols are available and the entropy encoder 1908 and the parser 30 are able to encode / decode the symbols into an encoded video sequence without loss, the entropy decoding portion of the decoder 1800, including the channel 1801, receiver 1802, buffer 1803, and parser 1804, may not be fully implemented in the local decoder 1906.

[0196] It can then be observed that any decoder technique other than parsing / entropy decoding, which exists in the decoder, must also exist in the corresponding encoder in essentially the same functional form. The description of encoder techniques can be simplified because encoder techniques are inverses of the fully described decoder techniques. More detailed descriptions are only required in certain areas, and are provided below.

[0197] During operation, the source encoder 1903 can perform motion-compensated predictive coding, i.e., predictively coding the input image with reference to one or more previously encoded images from the video sequence designated as "reference images". In this way, the coding engine 1907 encodes the differences between pixel blocks of the input image and pixel blocks of the reference image, which can be selected as the prediction reference for the input image.

[0198] The local video decoder 1906 can decode encoded video data of an image that can be designated as a reference image, based on symbols created by the source encoder 1903. The operation of the encoding engine 1907 can advantageously be a lossy process. When encoded video data is available in the video decoder (… Figure 19 When decoded at (not shown), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder 1906 replicates the decoding process, which can be performed by the video decoder on the reference image, and can store the reconstructed reference image in the reference image cache 1905. In this way, the encoder 1900 can locally store a copy of the reconstructed reference image that shares common content (no transmission errors) with the reconstructed reference image that will be obtained by the remote video decoder.

[0199] Predictor 1904 can perform a prediction search on encoding engine 1907. That is, for a new image to be encoded, predictor 1904 can search in reference image memory 1905 for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new image. Predictor 1904 can operate pixel-by-pixel based on sample blocks to find suitable prediction references. In some cases, as determined by the search results obtained by predictor 1904, the input image may have prediction references obtained from multiple reference images stored in reference image memory 1905.

[0200] The controller 1902 can manage the encoding operations of the video encoder 1903, including, for example, setting parameters and subgroup parameters for encoding video data.

[0201] The outputs of all the aforementioned functional units can be entropy encoded in the entropy encoder 1908. The entropy encoder performs lossless compression on the symbols generated by the various functional units using techniques known to those skilled in the art, such as Huffman coding, variable-length coding, and arithmetic coding, thereby converting the symbols into an encoded video sequence.

[0202] Transmitter 1909 can buffer the encoded video sequence created by entropy encoder 1908, thereby preparing it for transmission via communication channel 1911, which can be a hardware / software link to a storage device that will store the encoded video data. Transmitter 1909 can combine encoded video data from video encoder 1903 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0203] The controller 1902 manages the operation of the encoder 1900. During encoding, the controller 1902 can assign a specific encoded image type to each encoded image, but this may affect the encoding technique applicable to the corresponding image. For example, an image can typically be assigned to any of the following image types:

[0204] An intra-frame image (I-image) is an image that can be encoded and decoded without using any other images in the sequence as a prediction source. Some video codecs allow different types of intra-frame images, including, for example, Independent Decoder Refresh (IDR) images. Those skilled in the art will understand variations of I-images and their corresponding applications and characteristics.

[0205] A predictive image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and reference index to predict sample values ​​for each block.

[0206] A bidirectional predictive image (B-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses up to two motion vectors and a reference index to predict sample values ​​for each block. Similarly, multiple predictive images can use more than two reference images and associated metadata to reconstruct a single block.

[0207] The source image is typically spatially subdivided into multiple sample blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks), and each block is encoded sequentially. These blocks can be predictively coded with reference to other (already coded) blocks, which are determined by the coding assignment of the corresponding image applied to the block. For example, a block of an I-image can be non-predictively coded, or the block can be predictively coded (spatial prediction or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be predictively coded with reference to a previously coded reference image via spatial prediction or temporal prediction. Blocks of a B-image can be predictively coded with reference to one or two previously coded reference images via spatial prediction or temporal prediction.

[0208] The video encoder 1900 can perform encoding operations according to predetermined video coding techniques or standards, such as those specified in ITU-T H.265 Recommendation. In operation, the video encoder 1900 can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0209] In this embodiment, the transmitter 1909 may transmit additional data while transmitting encoded video. The source encoder 1903 may include such data as part of the encoded video sequence. Additional data may include temporal / spatial / SNR enhancement layers, redundant images and slices, other forms of redundant data, SEI messages, VUI parameter set fragments, etc.

[0210] Some embodiments may relate to systems, methods, and / or computer-readable media at any possible level of technical detail in the integration. The computer-readable medium may include a computer-readable non-transitory storage medium (or medium) having computer-readable program instructions thereon for causing a processor to perform operations.

[0211] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, such as punch cards, or mechanical encoding devices with raised structures in recesses on which instructions are recorded, and any suitable combination thereof. The computer-readable storage medium used in this document should not be construed as a transient signal, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses through fiber optic cables), or electrical signals propagating through wires.

[0212] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device. This network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0213] Computer-readable program code / instructions used to perform operations can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages. Programming languages ​​include object-oriented programming languages ​​such as Smalltalk and C++, and procedural programming languages ​​such as the "C" programming language or similar languages. Computer-readable program instructions can execute entirely on the user's computer, partially on the user's computer, or as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can connect to the user's computer via any type of network, including local area networks (LANs) or wide area networks (WANs), or can connect to an external computer (e.g., through the internet provided by an internet service provider). In some embodiments, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can execute computer-readable program instructions by utilizing state information of computer-readable program instructions to personalize the electronic circuits to perform multiple aspects or operations.

[0214] These computer-readable program instructions may be provided to the processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a manner for implementing the functions / actions specified in the flowcharts and / or block diagram blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium having the instructions stored thereon comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in the flowcharts and / or block diagram blocks.

[0215] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other equipment to cause a series of operational steps to be performed on the computer, other programmable apparatus or other equipment to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other equipment, perform the functions / actions specified in the flowchart and / or block diagram blocks.

[0216] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. The method, computer system, and computer-readable medium may include more blocks, fewer blocks, different blocks, or blocks arranged differently than those depicted in the figures. In some alternative implementations, the functions described in a module may differ in order from those shown in the figures. For example, depending on the functions involved, two consecutive blocks shown may actually be executed simultaneously or substantially simultaneously, or these blocks may sometimes be executed in reverse order. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0217] It is evident that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or combinations of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, this document describes the operation and behavior of the systems and / or methods without referring to any specific software code—it is understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0218] No element, action, or instruction used herein should be construed as critical or necessary unless explicitly described as such. Furthermore, as used herein, the terms “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more.” Additionally, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and may be used interchangeably with “one or more.” The term “an” or similar language is used if only one item is intended to be referred to. Furthermore, as used herein, the terms “including,” “having,” etc., are intended to indicate open-ended terms. Further, unless otherwise explicitly stated, the term “based on” is intended to mean “at least partially based on.”

[0219] Descriptions of various aspects and embodiments have been presented for illustrative purposes, but are not intended to be exhaustive or limiting of the disclosed embodiments. Even combinations of features recited in the claims and / or disclosed in the specification are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways not specifically stated in the claims and / or not disclosed in the specification. While each dependent claim listed below may be directly dependent on only one claim, the disclosure of possible implementations includes combinations of each dependent claim with every other claim in the claim set. Numerous modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, practical applications or improvements to techniques found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for video decoding, executed by at least one processor, the method comprising: Receive video data in Open Media Consortium Video 1 (AV1) format, the video data including data from at least two chroma prediction residual signal blocks; A transformation is performed between at least one signal block of a combination of at least two chromaticity prediction residual signal blocks, whose size is less than or equal to that of the at least two chromaticity prediction residual signal blocks. The chromaticity prediction residual signal block includes an N1xN2 Cb residual block and an N1xN2 Cr residual block. The at least one signal block includes a 2xN1xN2 three-dimensional (3-D) cube. The transformation includes: performing a 2-point 1-D transformation on each of the 2x1 vectors along the z-axis of the 3-D cube, and performing an N1xN2 transformation on the xy-plane of the 3-D cube; and The video data is decoded based on at least one signal block comprising a combination of at least two chroma prediction residual signal blocks with a size less than or equal to the aforementioned two signal blocks.

2. The method according to claim 1, characterized in that, The transformation includes: The coefficients of the at least one signal block are dequantized using an index notified by a signal, the size of the at least one signal block being less than or equal to the combination of the chromaticity prediction residual signal blocks, and The dequantized index, notified by the signal, is transformed by multiplying a 2xN1xN2 matrix by an LxM matrix. Where 2xN1xN2 is the size of the combination of the chromaticity prediction residual blocks, and Where LxM is the size of the combination of chromaticity prediction residual blocks that is less than or equal to the size of the chromaticity prediction residual blocks.

3. The method according to claim 2, characterized in that, The result of transforming the dequantized index notified by the signal by multiplying it by a 2xN1xN2 matrix LxM includes: a reconstructed 2xN1xN2 vector, and Decoding the video data includes: reconstructing the N1xN2 Cb residual block and the N1xN2 Cr residual block based on the reconstructed 2xN1xN2 vector.

4. The method according to claim 3, characterized in that, The at least one signal block includes: the interleaving of the at least two chromaticity prediction residual signal blocks, the interleaving being achieved by alternately using samples from the N1xN2Cb residual block and samples from the N1xN2Cr residual block.

5. The method according to claim 1, characterized in that, The transformation includes: performing an N1-point 1-D transformation on each of the N1x1 vectors along the x-axis of the 3-D cube, performing an N2-point 1-D transformation on each of the N2x1 vectors along the y-axis of the 3-D cube, and performing a 2-point transformation on each of the 2x1 vectors along the z-axis of the 3-D cube.

6. The method according to any one of claims 1 to 5, characterized in that, Performing the transformation includes performing a primary transformation and a secondary transformation.

7. The method according to claim 6, characterized in that, At least one of the primary transformation and the secondary transformation includes setting the coefficient to zero in response to a predetermined condition that determines the value of the coordinate associated with the coefficient of at least one of the chromaticity prediction residual signal blocks is greater than or equal to a threshold.

8. A video decoding apparatus, the apparatus comprising: At least one memory configured to store computer program code; At least one processor is configured to access and operate in accordance with the instructions of the computer program code, the computer program code comprising: The receiving code is configured to enable the at least one processor to receive video data in Open Media Consortium Video 1 (AV1) format, the video data including data of at least two chroma prediction residual signal blocks; The code is configured to perform a transformation between at least one signal block of a size less than or equal to the combination of the at least two chroma prediction residual signal blocks and the at least two chroma prediction residual signal blocks, wherein the chroma prediction residual signal blocks comprise: an N1xN2 Cb residual block and an N1xN2 Cr residual block, and the at least one signal block comprises: a 2xN1xN2 three-dimensional (3-D) cube, the transformation comprising: performing a 2-point 1-D transformation on each of the 2x1 vectors along the z-axis of the 3-D cube, and performing an N1xN2 transformation on the xy-plane of the 3-D cube; and The decoding code is configured to decode the video data based on at least one signal block comprising a combination of at least two chroma prediction residual signal blocks with a size less than or equal to the aforementioned two signal blocks.

9. The apparatus according to claim 8, characterized in that, The transformation includes: The coefficients of the at least one signal block are dequantized using an index notified by a signal, the size of the at least one signal block being less than or equal to the combination of the chromaticity prediction residual signal blocks, and The dequantized index, notified by the signal, is transformed by multiplying a 2xN1xN2 matrix by an LxM matrix. Where 2xN1xN2 is the size of the combination of the chromaticity prediction residual blocks, and Where LxM is the size of the combination of chromaticity prediction residual blocks that is less than or equal to the size of the chromaticity prediction residual blocks.

10. The apparatus according to claim 9, characterized in that, The result of transforming the dequantized index notified by the signal by multiplying it by a 2xN1xN2 matrix LxM includes: a reconstructed 2xN1xN2 vector, and Decoding the video data includes: reconstructing the N1xN2 Cb residual block and the N1xN2 Cr residual block based on the reconstructed 2xN1xN2 vector.

11. The apparatus according to claim 10, characterized in that, The at least one signal block includes: the interleaving of the at least two chromaticity prediction residual signal blocks, the interleaving being achieved by alternately using samples from the N1xN2Cb residual block and samples from the N1xN2Cr residual block.

12. The apparatus according to claim 8, characterized in that, The transformation includes: performing an N1-point 1-D transformation on each of the N1x1 vectors along the x-axis of the 3-D cube, performing an N2-point 1-D transformation on each of the N2x1 vectors along the y-axis of the 3-D cube, and performing a 2-point transformation on each of the 2x1 vectors along the z-axis of the 3-D cube.

13. The apparatus according to any one of claims 8 to 12, characterized in that, Performing the transformation includes performing a primary transformation and a secondary transformation.

14. A video decoding apparatus, the apparatus comprising: The receiving module is used to receive video data in Open Media Consortium Video 1 (AV1) format, the video data including data of at least two chroma prediction residual signal blocks; An execution module is configured to perform a transformation between at least one signal block (a combination of at least two chroma prediction residual signal blocks) and the at least two chroma prediction residual signal blocks, wherein the chroma prediction residual signal blocks comprise: an N1xN2 Cb residual block and an N1xN2 Cr residual block, and the at least one signal block comprises: a 2xN1xN2 three-dimensional (3-D) cube, the transformation comprising: performing a 2-point 1-D transformation on each of the 2x1 vectors along the z-axis of the 3-D cube, and performing an N1xN2 transformation on the xy-plane of the 3-D cube; and A decoding module is used to decode the video data based on at least one signal block comprising a combination of at least two chroma prediction residual signal blocks with a size less than or equal to the above two chroma prediction residual signal blocks.

15. A non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 7.

16. A computer device comprising a processor and a memory, wherein, The memory stores instructions that, when executed by the processor, cause the processor to implement the method according to any one of claims 1 to 7.

17. A method for processing video bitstreams, characterized in that, The video bitstream is decoded based on the video decoding method according to any one of claims 1 to 7.

Images