Cross-component prediction method and system, and computer-readable medium

Abstract

A system and method for cross-component prediction based on a deep neural network (DNN) are provided. The method comprises the steps of: inputting a reconstructed luminance block of an image or video into a DNN; and predicting a reconstructed chroma block of the image or video based on the input reconstructed luminance block. Luma and chroma reference information and additional information may also be input to the DNN to predict the reconstructed chroma block. Various inputs may be generated using processes such as downsampling and transformation.

Description

Technology Field

[0001] Cross-reference regarding related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 210,741 filed on June 15, 2021 and U.S. Application No. 17 / 749,730 filed on May 20, 2022.

[0003] Embodiments of the present invention relate to a method and system for DNN-based cross-component prediction. Background Technology

[0004] Traditional video coding standards such as H.264 / AVC (H.264 / Advanced Video Coding), HEVC (High-Efficiency Video Coding), and VVC (Versatile Video Coding) are designed within similar (recursive) block-based hybrid prediction / transform frameworks that optimize overall efficiency by intensively manualizing individual coding tools, such as intra / inter prediction, integer transformation, and context-adaptive entropy coding. Essentially, spatiotemporal pixel neighbors are utilized in constructing prediction signals to obtain corresponding residuals for subsequent transformation, quantization, and entropy coding. Meanwhile, the characteristic of Deep Neural Networks (DNNs) is to analyze spatiotemporal information within the receptive regions of neighboring pixels to extract spatiotemporal stimuli at various levels. The ability to explore high nonlinearity and non-local spatiotemporal correlations offers promising opportunities for significantly improved compression quality.

[0005] One objective of video coding and decoding may be to reduce redundancy in the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage requirements, sometimes by more than double orders of magnitude. Both lossless compression and lossy compression, as well as combinations thereof, can be employed. Lossless compression refers to a technique capable of reconstructing an exact copy of the original signal from the compressed original signal. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to be useful for the intended application. For video, lossy compression is employed. The amount of acceptable distortion may vary depending on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of television contribution applications. The achievable compression ratio can be higher as the acceptable / tolerated distortion increases.

[0006] Traditional encoders can achieve better compression performance by utilizing information from different components and other additional information to predict different components. However, the cross-component linear prediction mode in intra-prediction does not perform well compared to DNN-based methods. The characteristic of DNNs is their ability to extract different high-level stimuli, and their capacity to explore high non-linearity and non-local correlations offers promising opportunities for high compression quality. Embodiments of the present disclosure achieve superior compression performance by using a DNN-based model to process arbitrary shapes of the luminal component, reference component, and additional information to predict the reconstructed chroma component.

[0007] Embodiments of the present invention provide a Cross Component Prediction (CCP) model as a new mode of intra prediction using a Deep Neural Network (DNN). This model achieves superior compression performance by predicting chroma components using information provided by an encoder, such as lumina components, quantization parameter (QP) values, and block depth. Previous NN-based intra prediction approaches aim only for the prediction of lumina components or generate predictions for all three channels while ignoring the correlation between chroma components and other additional information.

[0008] According to an embodiment, a method is provided that is performed by at least one processor. The method comprises the steps of: acquiring a reconstructed luminance block of an image or video; inputting the reconstructed luminance block into a DNN; acquiring a reference component and additional information associated with the reconstructed luminance block; inputting the reference component and the additional information into the DNN; and the DNN predicting a reconstructed chroma block of the image or video based on the reconstructed luminance block, the reference component, and the additional information.

[0009] According to an embodiment, a system is provided. The system comprises at least one memory configured to store computer program code; and at least one processor configured to access the computer program code and operate as commanded by the computer program code. The computer program code comprises: an input code configured to cause the at least one processor to input a reconstructed luminal block of an image or video, additional information associated with the reconstructed luminal block, and a reference component to a deep neural network (DNN) implemented by the at least one processor; and a prediction code configured to cause the at least one processor to cause the DNN to predict a reconstructed chroma block of the image or video based on the input reconstructed luminal block, the reference component, and the additional information.

[0010] According to an embodiment, a computer-readable non-transient medium is provided for storing computer code. When the computer code is executed by at least one processor, the at least one processor is configured to implement a deep neural network (DNN); input a reconstructed luminal block of an image or video, additional information associated with the reconstructed luminal block, and a reference component into the deep neural network (DNN); and the DNN is configured to predict a reconstructed chroma block of the image or video based on the input reconstructed luminal block, the reference component, and the additional information. Brief explanation of the drawing

[0011] Additional features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. FIG. 2 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. FIG. 3 is a schematic diagram of a simplified block diagram of a decoder according to one embodiment. FIG. 4 is a schematic diagram of a simplified block diagram of an encoder according to one embodiment. FIG. 5 is a schematic diagram of a simplified block diagram of an input generation process according to one embodiment. FIG. 6 is a schematic diagram of a simplified block diagram of a cross-component prediction process according to one embodiment. FIG. 7 is a block diagram of computer code according to an embodiment. FIG. 8 is a drawing of a computer system suitable for implementing an embodiment of the present disclosure. Specific details for implementing the invention

[0012] FIG. 1 shows a simplified block diagram of a communication system (100) according to an embodiment of the present invention. The communication system (100) may include at least two terminals (110, 120) interconnected through a network (150). For unidirectional transmission of data, the first terminal (110) may encode video data at a local location to transmit to another terminal (120) through the network (150). The second terminal (120) may receive the coded video data from the other terminal from the network (150), decode the coded data, and display the restored video data. Unidirectional data transmission may be common in media that provide applications, etc.

[0013] FIG. 1 illustrates a second pair of terminals (130, 140) provided to support bidirectional transmission of coded video that may occur, for example, during a video conference. For bidirectional transmission of data, each terminal (130, 140) can code video data captured at a local location to transmit to another terminal via a network (150). Each terminal (130, 140) can also receive coded video data transmitted by the other terminal, decode the coded data, and display the recovered video data on a local display device.

[0014] In FIG. 1, terminals (110–140) may be exemplified as servers, personal computers, smartphones, and / or any other type of terminal. For example, terminals (110–140) may be laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (150) represents any number of networks that transmit coded video data between terminals (110–140), including, for example, wired and / or wireless communication networks. The communication network (150) may exchange data over circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of the network (150) may not be important to the operation of the present disclosure unless explicitly described below in the specification.

[0015] FIG. 2 illustrates the arrangement of a video encoder and a decoder in a streaming environment as an example of an application to the disclosed subject. The disclosed subject may be similarly applied to other video enabled applications, including, for example, the storage of compressed video on digital media including video conferencing, digital TV, CD, DVD, memory stick, etc.

[0016] As shown in FIG. 2, the streaming system (200) may include a capture subsystem (513) that may include a video source (201) and an encoder (203). The video source (201) may be, for example, a digital camera and may generate an uncompressed video sample stream (202). The uncompressed video sample stream (202) may provide a high data volume compared to an encoded video bitstream and may be processed by an encoder (203) coupled to the video source (201). The encoder (203) may include hardware, software, or a combination thereof that enables or implements aspects of the disclosed subject matter as described in more detail below. The encoded video bitstream (504) may have a lower data volume compared to the sample stream and may be stored in a streaming server (205) for later use. One or more streaming clients (206) can access the streaming server (205) to retrieve a video bitstream (209) which may be a copy of the encoded video bitstream (204).

[0017] In an embodiment, the streaming server (205) may also function as a Media-Aware Network Element (MANE). For example, the streaming server (205) may be configured to prune an encoded video bitstream (204) to potentially tailor a different bitstream to one or more streaming clients (206). In an embodiment, the MANE may be provided separately from the streaming server (205) in the streaming system (200).

[0018] A streaming client (206) may include a video decoder (210) and a display (212). The video decoder (210) may decode a video bitstream (209), which is, for example, an incoming copy of encoded video data (204), and generate an outgoing video sample stream (211) that can be rendered to a display (512) or another rendering device (not shown). In some streaming systems, the video bitstream (204, 209) may be encoded according to a specific video coding / compression standard. Examples of such standards may include, but are not limited to, ITU-T Recommendation H.265. A video coding standard under development is informally known as VVC (Versatile Video Coding). Embodiments of the present disclosure may be used in the context of VVC.

[0019] FIG. 3 shows an exemplary functional block diagram of a video decoder (610) according to one embodiment of the present disclosure.

[0020] The video decoder (210) may include a channel (312), a receiver (310), a buffer memory (315), an entropy decoder / parser (320), a scaler / inverse conversion unit (351), an intra-picture prediction unit (352), a motion compensation prediction unit (353), an aggregator (355), a loop filter unit (356), a reference picture memory (357), and a current picture memory. In at least one embodiment, the video decoder (210) may include an integrated circuit, a series of integrated circuits, and / or other electronic circuits. The video decoder (210) may also be implemented as software running on one or more CPUs having partially or wholly associated memory.

[0021] In this embodiment and in other embodiments, the receiver (310) may receive one or more coded video sequences at a time, which are decoded by the decoder (310), wherein the decoding of each coded video sequence is independent of the other coded video sequences. The coded video sequences may be received from a channel (312) which may be a hardware / software link to a storage device that stores the encoded video data. The receiver (310) may receive the encoded video data along with other data, e.g., coded audio data and / or ancillary data streams, which may be forwarded to each using an entity (not shown). The receiver (310) may separate the coded video sequences from the other data. To prevent network jitter, a buffer memory (315) may be coupled between the receiver (310) and the entropy decoder / parser (320) (hereinafter "parser"). When the receiver (310) receives data from a storage / forwarding device of sufficient bandwidth and controllability, or from an isosynchronous network, the buffer (315) may not be needed or may be small. For use in a best-effort packet network such as the Internet, the buffer memory (315) may be required, may be relatively large, and advantageously may be of an adaptive size.

[0022] The video decoder (210) may include a parser (320) to reconstruct symbols (321) from an entropy-coded video sequence. Categories of these symbols include information used to manage the operation of the video decoder (210) and potential information for controlling a rendering device, such as a display (212), which can be coupled to the decoder of the decoder as illustrated in FIG. 2. The control information for the rendering device(s) may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (320) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow video coding techniques or standards and may follow various principles well known to those skilled in the art, including variable-length coding, Huffman coding, and arithmetic coding with or without context sensitivity. The parser (320) can extract a set of subgroup parameters for at least one of the pixel subgroups from a coded video sequence based on at least one parameter corresponding to a subgroup. Subgroups may include a Group of Picture (GOP), picture, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), prediction unit (PU), etc. The parser (320) can also extract transform coefficients, quantizer parameter values, motion vectors, etc. from the coded video sequence information.

[0023] The parser (320) can generate symbols (321) by performing entropy decoding / parsing operations on a video sequence received from the buffer memory (315).

[0024] The reconstruction of the symbol (321) may be associated with a number of different units depending on the type of the coded video picture or a part thereof (e.g., inter- and intra-pictures, inter- and intra-blocks) and other factors. How a unit is associated may be controlled by subgroup control information parsed from the coded video sequence by the parser (320). The flow of this subgroup control information between the parser (320) and the number of units below it is not shown for brevity.

[0025] In addition to the functional blocks already mentioned, the video decoder (210) can be conceptually subdivided into a number of functional units as described below. In actual implementations operating under commercial constraints, many of these functional units may interact closely with one another and be integrated with one another, at least partially. However, for the sake of illustrating the subject matter disclosed, the conceptual subdivision into the functional units below is appropriate.

[0026] One unit may be a scaler / inverse transform unit (351). The scaler / inverse transform unit (351) may receive quantized transform coefficients as well as control information including the inverse transform to be used, block size, quantization factor, quantization scaling matrix, etc., as symbol(s) (321) from the parser (320). The scaler / inverse transform unit (351) may output a block containing sample values ​​that can be input to an aggregator (355).

[0027] In some cases, the output sample of the scaler / inverse conversion unit (351) may be related to an intra-coded block, that is, a block that can use prediction information from a previously reconstructed part of the current picture, without using prediction information from a previously reconstructed picture. This prediction information may be provided by an intra-picture prediction unit (352). In some cases, the intra-picture prediction unit (352) generates a block of the same size and shape as the block being reconstructed by using surrounding already reconstructed information retrieved from the current (partially reconstructed) picture from the current picture memory (358). In some cases, the aggregator (355) may add the prediction information generated by the intra-prediction unit (3652) to the output sample information provided by the scaler / inverse conversion unit (351) on a sample basis.

[0028] In other cases, the output samples of the scaler / inverse unit (351) may be intercoded and potentially related to a motion-compensated block. In such cases, the motion-compensated prediction unit (353) may access the reference picture memory (357) to retrieve samples used for inter-picture prediction. After motion-compensating the samples retrieved according to the symbol (3621) related to the block, these samples may be added to the output of the scaler / inverse unit (351) (referred to as residual samples or residual signals in this case) by the aggregator (355) to generate output sample information. The address in the reference picture memory from which the motion-compensated prediction unit retrieves the prediction samples may be controlled by a motion vector. The motion vector may be available to the motion-compensated unit (353) in the form of a symbol (321) that may have X, Y, and reference picture components, for example. Motion compensation may also include interpolation of sample values ​​drawn from reference picture memory (357) when the accurate motion vector of a subsample is used, a motion vector prediction mechanism, etc.

[0029] The output sample of the aggregator (355) may be subject to various loop filtering techniques in the loop filter unit (356). The video compression technique may include an in-loop filter technique that is controlled by parameters included in the coded bitstream and made available to the loop filter unit (356) as a symbol (321) from the parser (320), but may also respond to meta-information obtained while decoding a previous (in decoding order) part of the coded picture or coded video sequence, as well as respond to previously reconstructed and loop-filtered sample values.

[0030] The output of the loop filter unit (356) may be a sample stream that can be output to a render device such as a display (212) and may also be stored in a reference picture memory (357) for use in future inter-picture prediction.

[0031] Once a specific coded picture has been completely reconstructed, it can be used as a reference picture for later predictions. When a coded picture is completely reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (320)), the current reference picture may become part of the reference picture memory (357), and a new current picture memory may be reallocated before starting the reconstruction of the next coded picture.

[0032] The video decoder (210) may perform decoding operations according to a predetermined video compression technique that may be documented in a standard, such as ITU-T Rec. H.265. The coded video sequence may follow the syntax specified by the video compression technique or standard used in the sense that it complies with the syntax of the video compression technique or standard, particularly as specified in the profile document therein. Additionally, to comply with a given video compression technique or standard, the complexity of the coded video sequence may be within a range defined by a level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further limited by HRD specifications and metadata for managing the virtual reference decoder (HRD) buffer signaled in the coded video sequence.

[0033] In one embodiment, the receiver (310) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (210) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, a temporal layer, a spatial layer or an SNR enhancement layer, a redundant slice, a redundant picture, a forward error correction code, etc.

[0034] FIG. 4 shows an exemplary functional block diagram of a video encoder (203) associated with a video source (201) according to one embodiment of the present disclosure.

[0035] The video encoder (203) may include, for example, an encoder which is a source coder (430), a coding engine (432), a (local) decoder (433), a reference picture memory (434), a predictor (435), a transmitter (440), an encoder which is an entropy (445), a controller (450), and a channel (460).

[0036] The video encoder (203) can receive video samples from a video source (201) (not part of the encoder) capable of capturing video image(s) to be encoded by the video encoder (203).

[0037] A video source (201) may provide a source video sequence to be coded by a video encoder (203) in the form of a digital video sample stream, which may have any appropriate bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCb, RGB, ...), and any appropriate sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (201) may be a storage device capable of storing a pre-prepared video. In a video conferencing system, the video source (201) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures or images that impart motion when viewed sequentially. A picture itself may consist of a spatial array of pixels, and each pixel may contain one or more samples depending on the sampling structure, color space, etc. being used. A person skilled in the art will easily understand the relationship between pixels and samples. The following description focuses on the sample.

[0038] According to one embodiment, the video encoder (203) can encode and compress pictures of a source video sequence into a coded video sequence (443) in real time or under any other time constraint required by the application. Enforcing an appropriate coding rate is one function of the controller (450). The controller (450) can also control other functional units as described below and can be functionally coupled to such units. Such coupling is not shown for brevity. Parameters set by the controller (540) may include rate control related parameters (picture skip, quantizer, lambda value of rate distortion optimization technique...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. Since other functions of the controller (350) may be related to the video encoder (203) optimized for a specific system design, a person skilled in the art will be able to easily identify other functions of the controller (350).

[0039] Some video encoders operate in a manner that can be easily recognized by those skilled in the art as a "coding loop." As an oversimplified description, the coding loop may consist of the encoding portion of the source coder (430) (responsible for generating symbol and reference picture(s) based on the input picture to be coded), and a (local) decoder (433) embedded in the encoder (203) that reconstructs the symbol to generate sample data that the (remote) decoder will also generate if the compression between the symbol and the coded video bitstream is lossless in a particular video compression technique. The reconstructed sample stream may be input into the reference picture memory (434). Since the decoding of the symbol stream leads to a bit-exact result regardless of the decoder location (local or remote), the contents of the reference picture memory are also bit-exact between the local encoder and the remote encoder. In other words, the prediction part of the encoder "recognizes" as a reference picture sample exactly the same as what the decoder "sees" when using prediction during decoding. The basic principle of this reference picture synchronicity (and the resulting drift, for example, when synchronicity cannot be maintained due to channel error) is known to those skilled in the art.

[0040] The operation of the “local” decoder (433) may be the same as the operation of the “remote” decoder (210), which has already been described in detail in relation to FIG. 3. Since symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (445) and the parser (320) may be lossless, the entropy decoding portion of the decoder (310), including the channel (312), receiver (310), buffer (315), and parser (320), may not be entirely implemented in the local decoder (433).

[0041] What can be observed at this point is that any decoder technique, excluding parsing / entropy decoding present in the decoder, needs to exist in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject focuses on decoder operation. Descriptions of encoder techniques may be omitted as they are the opposite of the comprehensively described decoder techniques. Further details are required only in specific areas and are provided below.

[0042] As part of the operation, the source coder (430) may perform motion-compensated predictive coding to predictively code an input picture by referencing one or more previously coded pictures from a video sequence designated as a "reference picture." In this way, the coding engine (432) codes the difference between a pixel block of the reference picture(s) that can be selected as predictive reference(s) for the input picture and a pixel block of the input frame.

[0043] The local decoder (433) can decode the coded video data of a picture that can be designated as a reference picture based on the symbols generated by the source coder (430). The operation of the coding engine (432) may advantageously be a lossy process. If the coded video data can be decoded by a video decoder (not shown in FIG. 4), the reconstructed video sequence may generally be a replica of the source video sequence with some errors. The local decoder (433) can replicate the decoding process that can be performed by the video decoder on the reference picture and allow the reconstructed reference picture to be stored in the reference picture memory (434). In this way, the encoder (203) can locally store a copy of the reconstructed reference picture having common content as the reconstructed reference picture to be acquired by the far-end video decoder (no transmission errors).

[0044] The predictor (435) can perform a predictive search for the coding engine (432). That is, for a new picture to be coded, the predictor (435) can search the reference picture memory (434) for sample data (which is a candidate reference pixel block) that can serve as a suitable predictive reference for the new picture, or for specific metadata such as a reference picture motion vector, block shape, etc. The predictor (435) can operate on a sample block-by-pixel block basis to find a suitable predictive reference. In some cases, as determined by the search results obtained by the predictor (435), the input picture may have a predictive reference drawn from a number of reference pictures stored in the reference picture memory (434).

[0045] The controller (450) can manage the coding operations of the source coder (430), including, for example, the settings of parameters and subgroup parameters used to encode video data.

[0046] The output of all the aforementioned functional units may be subject to entropy coding in the entropy coder (445). The entropy coder converts the symbols generated by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable-length coding, and arithmetic coding.

[0047] The transmitter (440) may buffer the coded video sequence(s) generated by the entropy coder (445) and prepare for transmission through a communication channel (460), which may be a hardware / software link to a storage device for storing the encoded video data. The transmitter (440) may merge the coded video data from the source coder (430) with other data to be transmitted, e.g., coded audio data and / or an auxiliary data stream (source not shown).

[0048] The controller (450) can manage the operation of the video encoder (203). During coding, the controller (350) can assign a specific coded picture type to each picture, which can affect the coding technique that can be applied to each picture. For example, pictures can often be assigned as an Intra Picture (I picture), a Predictive Picture (P picture), and a Bi-directionally Predictive Picture (B picture):

[0049] An intra-picture (I-picture) may be one that can be coded and decoded without using any other picture within the sequence as a prediction source. Some video codecs allow different types of intra-pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of these variations of I-pictures and their respective applications and characteristics.

[0050] The prediction picture (P picture) may be coded and decoded using intra-prediction or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0051] A bidirectional predictive picture (Picture B) may be coded and decoded using intra-prediction or inter-prediction, which uses up to two motion vectors and reference indices to predict sample values ​​of each block. Similarly, a multiple-predictive picture may use metadata associated with more than two reference pictures for the reconstruction of a single block.

[0052] A source picture is generally spatially subdivided into multiple sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples, respectively) and can be coded on a block-by-block basis. Blocks can be predictively coded by referencing other (already coded) blocks as determined by the coding assignment applied to each picture in the block. For example, blocks in picture I can be non-predictively coded or predictively coded by referencing already coded blocks (spatial prediction or intra prediction) of the same picture. Pixel blocks in picture P can be predictively coded via spatial prediction or temporal prediction by referencing one previously coded reference picture. Blocks in picture B can be predictively coded via spatial prediction or temporal prediction by referencing one or two previously coded reference pictures.

[0053] The video encoder (203) can perform coding operations according to a predetermined video coding technique or standard, such as ITU-T Rec. H.265. In the operation, the video encoder (203) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Thus, the coded video data can follow the syntax specified by the video coding technique or standard used.

[0054] In one embodiment, the transmitter (440) may transmit additional data along with the encoded video. The video coder (430) may include such data as part of the encoded video sequence. The additional data may include other forms of redundancy data such as time / space / SNR enhancement layers, redundant pictures and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0055] Embodiments of the present disclosure provide DNN-based cross-component prediction. An exemplary embodiment is described below with reference to FIGS. 5 and FIGS. 6.

[0056] A video compression framework according to an embodiment of the present disclosure is described as follows. It is assumed that the input video contains a plurality of image frames equal to the total number of frames of the video. The frames are partitioned into spatial blocks, and each block may be repeatedly partitioned into smaller blocks. The blocks contain both a luminance component (510y) and a chroma component (520) including chroma channels (520u, 520t). During the intra-prediction process, the luminance component (510y) may be predicted first, and the two chroma channels (520u, 520t) may be predicted later. The prediction of the two chroma channels (520u, 520t) may be performed jointly or individually.

[0057] In one embodiment of the present disclosure, the reconstructed chroma component (520) is generated by a DNN-based model in both the encoder and the decoder, or only in the decoder. Two chroma channels (520u, 520t) may be generated together with a single network or separately from different networks. For each chroma channel, the chroma channel may be generated using a different network based on the block size. Signal processing, spatial or temporal filtering, scaling, weighted averaging, up / down sampling, pooling, recursive processing using memory, linear system processing, non-linear system processing, neural network processing, deep learning-based processing, AI processing, pre-trained network processing, machine learning-based processing, or a combination thereof may be used as modules in the embodiments of the present disclosure for DNN-based cross-component prediction. To process the reconstructed chroma component (520), one reconstructed chroma channel (e.g., one of chroma channels 520u and 520t) can be used to generate another reconstructed chroma channel (e.g., the other of chroma channels 520u and 520t).

[0058] According to an embodiment of the present disclosure, a DNN-based cross-component prediction model can be provided to improve the compression performance of the reconstructed chroma channels (520u, 520t) of a block based on the reconstructed luminance component (510y) of the block, the reference component, and other additional information provided by the encoder. According to an embodiment, 4:2:0 can be used for subsampling the chroma channels (520u, 520t). Thus, the chroma channels (520u, 520t) have lower resolution than the luminance component (510y).

[0059] Referring to FIG. 5, a process (500) is described below. The process (500) includes a workflow for generating input samples (580) for training and / or prediction in a general hybrid video coding system according to an embodiment of the present disclosure.

[0060] The reconstructed luminance component (510y) may be a luminance block that is 2N×2M block, where 2N is the width of the luminance block and 2M is the height of the luminance block. According to the embodiment, a first luminance reference (512y) that is 2N×2K block and a second luminance reference (514y) that is 2K×2M block may also be provided, where 2K represents the number of rows or columns in the luminance reference. To make the luminance size equal to the predicted output size, a downsampling process (591) is applied to the luminance component (510y), the first luminance reference (512y), and the second luminance reference (514y). The downsampling process (530) may be a traditional method such as bicubic and bilinear, or an NN-based downsampling method. After downsampling, the luminance component (510y) can become a downsampled luminance component (530y) having an N×M block size, the first luminance reference (512y) can become a downsampled first luminance reference (532y) having an N×K block size, and the second luminance reference (514y) can become a downsampled second luminance reference (534y) having a K×M block size. The downsampled first luminary reference (532y) and the downsampled second luminary reference (534y) can be converted (in step 592) to become the first converted luminary reference (552y) and the second converted luminary reference (554y), respectively, which match the size (also called the luminary block) of the downsampled luminary component (530y), and the first converted luminary reference (552y), the second converted luminary reference (554y), and the downsampled luminary component (530y) can be connected together (in step 592). For example, the conversion can be performed by duplicating the values ​​of the downsampled first luminary reference (532y) and the downsampled second luminary reference (534y) multiple times until their size becomes equal to the output block size (e.g., the size of the downsampled luminary component (530y)).

[0061] To predict the chroma component (520), adjacent references of the chroma component (520) (e.g., a first chroma reference (522) and a second chroma reference (524)) may also be added as optional references to generate a better chroma component. Referring to FIG. 5, the chroma component (520) may be a block of size N×M, which is a reconstructed chroma block that can be generated / predicted in the embodiments of the present disclosure. The chroma component (520) has two chroma channels (520u, 520t), and both channels (520u, 520t) may be used jointly. The first chroma reference (522) and the second chroma reference (524) may be obtained (at step 593) and may each have block sizes of N×K and K×M. According to the embodiment, the first chroma reference (522) and the second chroma reference (524) can be acquired twice, each corresponding to two chroma channels (520u, 520t). The first chroma reference (522) and the second chroma reference (524) can be converted (in step 594) into a first converted chroma reference (542) and a second converted chroma reference (544), each matching an N×M size. All image-based information (e.g., downsampled luminance component (530y), first transformed luminance reference (552y), second transformed luminance reference (554y), first transformed chroma reference (542), and second transformed chroma reference (544)) is combined (in step 595) to obtain input samples (580) for training the DNN and / or for prediction using the DNN. In addition to the luminance and chroma components, additional information may be added to the input for training and / or prediction of the neural network.For example, QP values ​​and block partition depth information can be used to generate an N×M size feature map and can be combined with an image-based feature map (e.g., downsampled luminance component (530y), first transformed luminance reference (552y), second transformed luminance reference (554y), first transformed chroma reference (542) and second transformed chroma reference (544)) (at step 595) to generate input samples (580) for training and / or prediction.

[0062] Referring to Fig. 6, the workflow of the process (600) in a typical hybrid video coding system is described below.

[0063] Since a set of reconstructed luminar block (610) (also called a luminar component), additional information (612), a luminar reference (614) adjacent to the luminar block (610), and a chroma reference (616) adjacent to the chroma block to be reconstructed can be used as input to the DNN (620), the model of the embodiment of the present invention can perform both training and prediction. The output (630) of the DNN (620) may be a predicted chroma component, and two chroma channels may be predicted using different DNN models or the same DNN model.

[0064] According to the embodiment, the input to the DNN (620) may be the input sample (580) described with reference to FIG. 5. For example, the reconstructed luminance block (610) may be a downsampled luminance component (530y), the adjacent luminance reference (614) may be one or more of the first transformed luminance reference (552y) and the second transformed luminance reference (554y), and the adjacent chroma reference (616) may be one or more of the first transformed chroma reference (542) and the second transformed chroma reference (544) (for one or both of the chroma channels (520u, 520t)). According to the embodiment, additional information may include, for example, QP values ​​and block partition depth information.

[0065] The combination, connection, or order of the method in which the reconstructed luma block (610), additional information (612), adjacent luma reference (614), and adjacent chroma reference (616) are used as inputs may vary. According to an embodiment, the additional information (612), adjacent luma reference (614), and / or adjacent chroma reference (616) may be optional inputs to the DNN (620) based on determination(s) by the coding system of the embodiment of the present disclosure.

[0066] According to an embodiment, the coding system of the present invention can calculate reconstruction quality by comparing, for example, the output (630) of a DNN (620) (e.g., predicted chroma component) with the original chroma block (660) and by comparing one or more chroma blocks from other prediction modes with the original chroma block (660) (step 650) (step 640). Based on the determination that one of the output (630) (e.g., predicted chroma component) and one or more chroma blocks from other prediction modes (step 650) has the highest reconstruction quality (e.g., closest to the original chroma block (660)), such a block (or mode) may be selected by the coding system to become the reconstructed chroma block (670).

[0067] According to an embodiment, at least one processor and memory for storing computer program instructions may be provided. When computer program instructions are executed by at least one processor, they may implement a system that performs any number of functions described in the present disclosure. For example, referring to FIG. 7, at least one processor may implement a system (700). The system (700) may include DNN(s) and at least one model of a DNN. Computer program instructions may include, for example, DNN code (710), input generation code (720), input code (730), prediction code (740), reconstruction quality code (750), and image acquisition code (760).

[0068] The DNN code (710) may be configured to enable at least one processor to implement the DNN(s) (and the model thereof) according to an embodiment of the present disclosure.

[0069] The input generation code (720) may be configured to cause at least one processor to generate an input for the DNN(s) according to an embodiment of the present disclosure (e.g., see description of FIG. 5). For example, the input generation code (720) may cause the process described with reference to FIG. 5 to be performed.

[0070] The input code (730) may be configured to allow at least one processor to input an input to the DNN(s) according to an embodiment of the present disclosure (e.g., see description of input to the DNN (620) shown in FIG. 6). For example, referring to FIG. 6, the input may include a reconstructed lumens block (610), additional information (612), a lumens reference (614) and / or a chroma reference (616).

[0071] According to an embodiment of the present disclosure, the prediction code (740) may be configured to allow at least one processor to predict a reconstructed chroma block by the DNN(s) (e.g., see description of the output (630) shown in FIG. 6).

[0072] According to an embodiment of the present disclosure, the reconstruction quality code (750) may be configured to allow at least one processor to calculate the reconstruction quality of a reconstructed chroma block predicted by a DNN and other reconstructed chroma block(s) predicted using other prediction mode(s) (e.g., see description of steps 640 and 650 illustrated in FIG. 6).

[0073] According to an embodiment of the present disclosure, the image acquisition code (760) may be configured to allow at least one processor to acquire an image using a reconstructed chroma block predicted by a DNN, or other reconstructed chroma block(s) predicted using other mode(s) (e.g., see description of step 640 and reconstructed chroma block (670) illustrated in FIG. 6). For example, the image acquisition code (760) may be configured to allow at least one processor to select one of a reconstructed chroma block and other reconstructed chroma blocks based on having the highest calculated reconstruction quality, and to acquire an image using such reconstructed chroma. According to an embodiment, the image acquisition code (760) may be configured to allow at least one processor to acquire an image using a reconstructed chroma block predicted by a DNN without a reconstruction quality used for selection among the calculated and / or reconstructed chroma blocks. According to an example, the reconstructed luma block can also be used to acquire an image.

[0074] Compared to previous cross-component prediction methods in intra-prediction mode, the embodiments of the present disclosure provide various advantages. For example, the embodiments of the present disclosure provide a flexible and general framework that accommodates reconstructed blocks of various shapes. Furthermore, the embodiments of the present disclosure include aspects that improve coding efficiency by optimizing the learning capabilities of a DNN model through the utilization of a transformation mechanism having various input information. Additionally, additional information can be used in conjunction with the DNN to improve prediction results.

[0075] The technology of the embodiments of the present disclosure described above may be implemented as computer software using computer-readable instructions and may be physically stored on one or more computer-readable media. For example, FIG. 8 illustrates a computer system (900) suitable for implementing embodiments of the subject matter disclosed.

[0076] Computer software may be coded using any suitable machine code or computer language, and may generate code containing instructions that can be executed directly or through interpretation, micro-code execution, etc., via assembly, compilation, linking, or similar mechanisms by a computer central processing unit (CPU), graphics processing unit (GPU), etc.

[0077] The command can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0078] The components of the computer system (900) illustrated in FIG. 8 are essentially exemplary and are not intended to imply any limitation to the scope of use or function of the computer software implementing the embodiments of the present disclosure. The configuration of the components should not be interpreted as having any dependency or requirement related to any one or combination of the components shown in the exemplary embodiments of the computer system (900).

[0079] The computer system (900) may include a specific human interface input device. This human interface input device may respond to input by one or more human users, for example, tactile input (e.g., keystroke, swip, data glove movement), audio input (e.g., voice, clap), visual input (e.g., gesture), and olfactory input (not shown). The human interface device may also be used to capture specific media that are not necessarily directly related to conscious input by a person, such as audio (e.g., voice, music, ambient sound), images (e.g., scanned images, picture images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0080] The input human interface device may include one or more of a keyboard (901), a mouse (902), a trackpad (903), a touch screen (910), a data glove, a joystick (905), a microphone (906), a scanner (907), and a camera (908) (each only one is shown).

[0081] The computer system (900) may also include a specific human interface output device. This human interface output device may stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. This human interface output device may include a tactile output device (e.g., tactile feedback via a touch screen (910), data gloves, or a joystick (905), but there may also be a tactile feedback device that does not serve as an input device). For example, the device may include an audio output device (e.g., a speaker (909), headphones (not shown)), a visual output device (e.g., a screen (910) including a CRT screen, an LCD screen, a plasma screen, and an OLED screen, each with or without touch screen input capability and with or without haptic feedback capability—some of which may be capable of two-dimensional visual output or three-dimensional or higher output through means such as stereographic output, virtual-reality glasses (not shown), a holographic display and a smoke tank (not shown)), and a printer (not shown).

[0082] The computer system (900) may also include human-accessible storage devices and associated media, such as optical media including a CD / DVD ROM RW (920) having a CD / DVD media (921), a thumb drive (922), a removable hard drive or solid-state drive (923), legacy magnetic media such as tape and floppy disk (not shown), and specialized ROM / ASIC / PLD-based devices such as a security dongle (not shown).

[0083] Those skilled in the art should also understand that the term “computer-readable medium” as used in connection with the subject matter disclosed herein does not include a transmitting medium, a carrier wave, or other transient signals.

[0084] The computer system (900) may also include an interface to one or more communication networks. The network may be, for example, a wireless, wired, optical network. The network may also be a local, wide-area, metropolitan, automotive and industrial, real-time, latency-tolerant network. Examples of networks include cellular networks including Ethernet, wireless LAN, GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, automotive and industrial networks including CANbus, etc. A specific network generally requires an external network interface adapter attached to a specific general-purpose data port or peripheral bus (949) (e.g., a USB port of the computer system (900)); others are generally integrated into the core of the computer system (900) by attaching to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (900) can communicate with other networks. Such communication may be unidirectional, receive-only (e.g., TV broadcasting), unidirectional transmit-only (e.g., from a CANbus to a specific CANbus device), or bidirectional (e.g., to another computer system using a local or wide-area digital network). Such communication may include communication to a cloud computing environment (955). Specific protocols and protocol stacks may be used for each of the networks and network interfaces as described above.

[0085] The aforementioned human interface device, human-accessible storage device, and network interface (954) can be attached to the core (940) of the computer system (900).

[0086] The core (940) may include one or more central processing units (CPUs) (941), graphics processing units (GPUs) (942), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (943), hardware accelerators (944) for specific tasks, etc. Along with read-only memory (ROM) (945), random access memory (946), and internal mass storage devices (947) such as internal hard drives or SSDs that are not accessible to the user, these devices may be connected via a system bus (948). In some computer systems, the system bus (948) may be accessible in the form of one or more physical plugs that enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus (948) or via a peripheral bus (949). Architectures for peripheral buses include PCI, USB, etc. A graphics adapter (950) may be included in the core (940).

[0087] The CPU (941), GPU (942), FPGA (943), and accelerator (944) can be combined to execute specific instructions that can construct the aforementioned computer code. The computer code may be stored in ROM (945) or RAM (946). Transitional data may also be stored in RAM (946), while permanent data may be stored, for example, in an internal mass storage device (947). Fast storage and retrieval of any of the memory elements may be made possible through the use of a cache memory that may be closely associated with one or more CPUs (941), GPUs (942), mass storage devices (947), ROM (945), RAM (946), etc.

[0088] A computer-readable medium may have computer code for performing various computer-implemented operations. The medium and the computer code may be specially designed and constructed for the purposes of this disclosure, or they may be of a type well known and available to those skilled in the art of computer software.

[0089] As an example, not limited to, the architecture (900), specifically the core (940), may provide functionality as a result of processor(s) (including CPU, GPU, FPGA, accelerator, etc.) executing software implemented on one or more types of computer-readable media. Such computer-readable media may be media associated with a mass storage device accessible to a user as described above, as well as specific storage devices of the core (940) of a non-transient nature, such as a mass storage device (947) or ROM (945) within the core. Software implementing various embodiments of the present disclosure may be stored on such devices and executed by the core (940). The computer-readable media may include one or more memory elements or chips as needed. Software may enable the core (940) and, in particular, the internal processor (including a CPU, GPU, FPGA, etc.) to execute a specific process or a specific part of a specific process described herein, including defining data structures stored in RAM (946) and modifying such data structures according to a process defined by the software. Additionally or alternatively, the computer system may provide a function that is otherwise implemented in a circuit (e.g., accelerator (944)) as a result of logic hardwired, which can operate instead of or with the software to execute a specific process or a specific part of a specific process described herein. References to software may include logic, and where appropriate, vice versa. References to computer-readable media may include circuits that store software for execution (e.g., integrated circuits (ICs)), circuits that implement logic for execution, or both, where appropriate. The present disclosure includes any suitable combination of hardware and software.

[0090] Although the present disclosure describes some non-limiting exemplary embodiments, there are modifications, permutations, and various alternative equivalents that fall within the scope of the present disclosure.

Claims

Claim 1 A cross-component prediction method performed by at least one processor, comprising: acquiring a reconstructed luminance block of an image or video; inputting the reconstructed luminance block into a deep neural network (DNN); acquiring a reference component and additional information associated with the reconstructed luminance block; and inputting the reference component and the additional information into the DNN. A cross-component prediction method comprising the step of the DNN predicting a reconstructed chroma block of an image or video based on the reconstructed lumina block, the reference component, and the additional information, wherein the reconstructed chroma block comprises a first chroma channel and a second chroma channel, the reference component comprises at least one of an adjacent lumina reference of the reconstructed lumina block and an adjacent chroma reference of the reconstructed chroma block to be predicted, and the additional information comprises at least one of a quantization parameter (QP) value and block division depth information, and the step of predicting the reconstructed chroma block comprises the step of predicting a first chroma channel of the reconstructed chroma block and predicting a second chroma channel of the reconstructed chroma block based on the first chroma channel. Claim 2 A cross-component prediction method according to claim 1, wherein the predicting step further comprises the step of the DNN predicting the reconstructed chroma block based on at least one of the reconstructed chroma block, the input adjacent chroma reference, and the adjacent chroma reference. Claim 3 A cross-component prediction method according to claim 1, further comprising the step of generating a feature map based on the additional information and connecting the generated feature map with another image-based feature map for DNN training. Claim 4 A cross-component prediction method according to claim 1, further comprising: a step of calculating the reconstruction quality of the reconstructed chroma block predicted by the DNN and the reconstruction quality of another reconstructed chroma block predicted using another prediction mode; and a step of acquiring the image or video using one of the reconstructed chroma block predicted by the DNN and the other reconstructed chroma block predicted using the other prediction mode, based on the highest of the calculated reconstruction qualities. Claim 5 A cross-component prediction method according to claim 1, further comprising the step of generating an input to the DNN, wherein the predicting step comprises the step of the DNN predicting a reconstructed chroma block of the image or video based on the input, and wherein the generating step comprises the step of reconstructing a lumina block and obtaining an adjacent lumina reference of the lumina block; the step of downsampling the lumina block to obtain the reconstructed lumina block as one of the inputs; the step of downsampling the adjacent lumina reference of the lumina block; and the step of converting the downsampled adjacent lumina reference to have the same size as the downsampled lumina block, and wherein the input to the DNN comprises a downsampled lumina block and a converted adjacent lumina reference. Claim 6 A method for predicting cross-components according to claim 5, wherein the luma block is a 2N×2M block, the adjacent luma reference includes a 2N×2K first luma reference block and a 2K×2M second luma reference block, N, K, and M are integers, 2N is the width, 2M is the height, and 2K is the number of rows or columns of the luma reference. Claim 7 A method for predicting cross components according to claim 6, wherein the size of the reconstructed luma block obtained by downsampling the luma block is N×M, and after downsampling the adjacent luma reference, the size of the first luma reference block is N×K and the size of the second luma reference block is K×M. Claim 8 A cross-component prediction method according to claim 5, wherein the generating step further comprises the step of converting adjacent chroma references of the reconstructed chroma block to be predicted to have the same size as the downsampled luminance block, and the input of the DNN comprises the downsampled luminance block, the converted adjacent luminance references, and the converted adjacent chroma references. Claim 9 A cross-component prediction system comprises: at least one memory configured to store computer program code; and at least one processor configured to access said computer program code and operate as commanded by said computer program code, wherein the computer program code is an input code configured to cause said at least one processor to input a reconstructed luminance block of an image or video, additional information associated with said reconstructed luminance block, and a reference component into a deep neural network (DNN) implemented by said at least one processor; A cross-component prediction system comprising: a prediction code configured to cause at least one processor to predict a reconstructed chroma block of an image or video based on the input reconstructed lumina block, the reference component, and the additional information, wherein the reconstructed chroma block comprises a first chroma channel and a second chroma channel, wherein the reference component comprises at least one of an adjacent lumina reference of the reconstructed lumina block and an adjacent chroma reference of the reconstructed chroma block to be predicted, and wherein the additional information comprises at least one of a quantization parameter (QP) value and block division depth information, and wherein the prediction of the reconstructed chroma block comprises predicting a first chroma channel of the reconstructed chroma block and predicting a second chroma channel of the reconstructed chroma block based on the first chroma channel. Claim 10 A cross-component prediction system according to claim 9, wherein the prediction code is further configured to allow the at least one processor to cause the DNN to predict the reconstructed chroma block based on the reconstructed chroma block and at least one of the input adjacent chroma reference and adjacent chroma reference. Claim 11 A cross-component prediction system according to claim 9, wherein the computer program code further comprises an input generation code configured to cause the at least one processor to generate an input to the DNN, and the prediction code further comprises causing the at least one processor to cause the DNN to predict a reconstructed chroma block of the image or video based on the input, and the input generation code comprises causing the at least one processor to generate an input to the DNN by causing the at least one processor to reconstruct a lumina block and obtain an adjacent lumina reference of the lumina block; downsample the lumina block to obtain the reconstructed lumina block as one of the inputs; downsample the adjacent lumina reference of the lumina block; and convert the downsampled adjacent lumina reference to have the same size as the downsampled lumina block. The input to the DNN comprises a downsampled lumina block and a converted adjacent lumina reference. Claim 12 A computer-readable, non-transient medium for storing computer code, wherein, when the computer code is executed by at least one processor, the at least one processor enables the implementation of a deep neural network (DNN); and inputs a reconstructed luminal block of an image or video, additional information associated with the reconstructed luminal block, and a reference component into the deep neural network (DNN); A computer-readable non-transient medium, wherein the above DNN is configured to predict a reconstructed chroma block of an image or video based on the input reconstructed lumina block, the reference component, and the additional information, wherein the reconstructed chroma block includes a first chroma channel and a second chroma channel, wherein the reference component includes at least one of an adjacent lumina reference of the reconstructed lumina block and an adjacent chroma reference of the reconstructed chroma block to be predicted, and wherein the additional information includes at least one of a quantization parameter (QP) value and block division depth information, and wherein the prediction of the reconstructed chroma block includes predicting a first chroma channel of the reconstructed chroma block and predicting a second chroma channel of the reconstructed chroma block based on the first chroma channel. Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete

Images