Content-Adaptive Online Training for DNN-Based Cross-Component Prediction with Scaling

Abstract

A method and apparatus for neural network-based cross-component prediction with a scaling factor during encoding or decoding of an image frame or video sequence may include: training a deep neural network (DNN) cross-component prediction (CCP) model with at least one or more scaling factors, where the at least one or more scaling factors are learned by optimizing a rate-distortion loss based on an input video sequence including a luma component; and for chroma prediction, reconstructing chroma components based on the luma component using the trained DNN CCP model with the at least one or more scaling factors. The trained DNN CCP model may be updated for chroma prediction of the input video sequence with the one or more scaling factors, and performing chroma prediction of the input video sequence with the updated DNN CCP model with the one or more scaling factors.

Description

[Technical field]

[0001] [Related Applications] This application claims priority to U.S. Provisional Application No. 63 / 210,762, filed June 15, 2021, and U.S. Patent Application No. 17 / 825,339, filed May 26, 2022, which are incorporated herein by reference in their entireties.

[0002] [Technical field] SUMMARY OF THE DISCLOSURE Embodiments of the present disclosure relate to neural network based cross-component prediction during encoding or decoding of images and / or video sequences. [Background technology]

[0003] Video coding and decoding reduces redundancy in an input video signal through compression. Compression, both lossless and lossy, can help reduce bandwidth or storage space requirements, in some cases by more than two orders of magnitude. Lossless compression refers to techniques where an exact copy of the original signal can be reconstructed from the compressed original signal. With lossy compression, the reconstructed signal is not identical to the original signal, but the distortion between the original and reconstructed signals is small enough to produce a reconstructed signal that is useful for the intended application. Lossy compression is widely used in video encoding or decoding. The amount of distortion tolerance may depend on the application. For example, a user of a particular consumer streaming application may tolerate higher distortion than a user of a television posting application.

[0004] Traditional video coding standards such as H.264 / Advanced Video Coding (H.264 / AVC), High-Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC) are designed on a similar (recursive) block-based hybrid prediction / transform framework, where individual coding tools such as intra / inter prediction, integer transform, and context-adaptive entropy coding are thoroughly handcrafted to optimize the overall efficiency. Essentially, the spatio-temporal pixel neighborhood is utilized in the prediction signal construction to obtain the corresponding residual for subsequent transform, quantization, and entropy coding. However, this approach cannot extract different levels of spatio-temporal stimuli by analyzing the spatio-temporal information of various layers. Therefore, for better compression efficiency and better compression quality, a device is needed that explores nonlinearity and non-local spatio-temporal correlation. Summary of the Invention

[0005] According to an aspect of the present disclosure, a method for neural network (NN) based cross-component prediction with scaling factor during encoding or decoding may be provided, the method comprising: training a deep neural network (DNN) cross-component prediction (CCP) model with at least one or more scaling factors, the at least one or more scaling factors being learned by optimizing a rate-distortion loss based on an input video sequence including a luma component; reconstructing chroma components based on the luma component using the trained DNN CCP model at the at least one or more scale factors for chroma prediction; updating the trained DNN CCP model for chroma prediction of the input video sequence with the one or more scaling factors; and performing chroma prediction of the input video sequence using the updated DNN CCP at the one or more scale factors.

[0006] According to an aspect of the present disclosure, an apparatus may be provided for scaling-based neural network (NN)-based cross-component prediction during encoding or decoding, the apparatus comprising: at least one memory configured to store program code; at least one processor configured to read the program code and to operate as directed by the program code; may include: The program code comprises: training code configured to cause the at least one processor to train a deep neural network (DNN) cross-component prediction (CCP) model with at least one or more scaling factors, the at least one or more scaling factors being learned by optimizing a rate-distortion loss based on an input video sequence including a luma component; and reconstructing code that causes the at least one processor to reconstruct chroma components based on the luma component using the trained DNN CCP model at the at least one or more scale factors for chroma prediction; update code for updating the trained DNN CCP model for chroma prediction of the input video sequence with the one or more scale factors; and executable code for performing chroma prediction of the input video sequence using the updated DNN CCP at the one or more scale factors. may include:

[0007] According to an aspect of the present disclosure, a non-transitory computer-readable medium for scaling-based neural network (NN) cross-component prediction during encoding or decoding may be provided, the non-transitory computer-readable medium comprising: training a deep neural network (DNN) cross-component prediction (CCP) model with at least one or more scaling factors, the at least one or more scaling factors being learned by optimizing a rate-distortion loss based on an input video sequence including a luma component; reconstructing chroma components based on the luma component using the trained DNN CCP model at the at least one or more scale factors for chroma prediction; updating the trained DNN CCP model for chroma prediction of the input video sequence with the one or more scaling factors; and performing chroma prediction of the input video sequence using the updated DNN CCP at the one or more scale factors. [Brief description of the drawings]

[0008] Further features, characteristics, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

[0009] [Figure 1] FIG. 1 is a simplified block diagram of a communication system according to an embodiment.

[0010] [Diagram 2] 2 is a block diagram of example components of one or more devices of FIG. 1.

[0011] [Diagram 3] FIG. 1 illustrates an example deep neural network (DNN) based cross-component prediction with scaling factor during encoding or decoding, according to an embodiment.

[0012] [Figure 4] 1 illustrates a flowchart of a method for deep neural network (DNN) based cross-component prediction (CCP) with scaling during encoding or decoding, according to an embodiment.

[0013] [Diagram 5] FIG. 1 is a diagram of a streaming environment according to an embodiment.

[0014] [Figure 6] FIG. 2 is a block diagram of a video decoder according to an embodiment.

[0015] [Figure 7] FIG. 2 is a block diagram of a video encoder according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] As mentioned above, conventional methods may utilize the neighborhood of spatio-temporal pixels for predictive signal construction to obtain the corresponding residual for subsequent transform, quantization, and entropy coding. However, this approach cannot extract different levels of spatio-temporal stimulation by analyzing the spatio-temporal information of different layers. Therefore, for better compression efficiency and better compression quality, a device is needed that explores nonlinearity and non-local spatio-temporal correlation.

[0017] By utilizing information from different components and additional side information, non-neural network based encoders can predict other components and achieve better compression performance. However, their performance is limited by Neural Networks For example, the cross-component linear prediction mode of intra prediction fails to perform as efficiently as methods based on deep neural networks (DNNs).

[0018] DNNs are essentially programmed to extract different levels of stimuli and have the ability to explore highly nonlinear and nonlocal correlations, which offers a promising opportunity to achieve high compression quality.

[0019] According to an embodiment of the present disclosure, a content-adaptive cross-component prediction method may be provided that improves compression performance by training online with a scaling factor. The online training may include training one or more models. According to an embodiment of the present disclosure, the DNN-based encoder and / or decoder may be partially updated during or at a specific time after estimation to provide better compression performance for one or more reconstructed components optimized for the input video.

[0020] FIG. 1 illustrates a simplified block diagram of a communication system (100) according to an embodiment of the present disclosure. The communication system 100 may include at least two terminals 140-130 interconnected via a network 150. In a unidirectional transmission of data, a first terminal (140) may code video data locally for transmission to another terminal (130) via the network (150). The second terminal (130) may receive the coded video data of the other terminal from the network (150), decode the coded data, and display the recovered video data. Unidirectional data transmission may be common in media serving applications, etc.

[0021] 1 illustrates a second pair of terminals (110, 120) adapted to support bidirectional transmission of coded video, such as may occur during a video conference. In the bidirectional transmission of data, each terminal (110, 120) may code locally captured video data for transmission to the other terminal over the network (150). Each terminal 110, 120 may also receive coded video data transmitted by the other terminal, decode the coded data, and display the recovered video data on a local display device.

[0022] In FIG. 1, the terminal devices 140-120 may be depicted as servers, personal computers, and smartphones, although the principles of the present disclosure are not limited thereto. The embodiments of the present disclosure may be applied with laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. The network (150) represents any number of networks that carry coded video data between the terminals (140-120), including, for example, wired and / or wireless communication networks. The communication network (150) may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include electronic communication networks, local area networks, wide area networks, and / or the Internet. For purposes of the present discussion, the architecture and topology of the network (150) may not be important to the operation of the present disclosure, unless otherwise noted below.

[0023] FIG. 2 is a block diagram of example components of one or more devices of FIG.

[0024] The device (200) may correspond to any one of the terminals (110 to 140). As shown in Fig. 2, the device (200) may include a bus (210), a processor (220), a memory (230), a storage component (240), an input component (250), an output component (260), and a communication interface (270).

[0025] The bus (210) includes components that enable communication between the components of the device (200). The processor (220) is implemented in hardware, firmware, or a combination of hardware and software. The processor (220) is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or another type of processing component. In some embodiments, the processor (220) includes one or more processors that are programmable to perform functions. The memory (230) includes random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by the processor (220).

[0026] The storage component 240 stores information and / or software related to the operation and use of the device 200. For example, the storage component 240 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cartridge, a magnetic tape, and / or another type of non-transitory computer-readable medium along with a corresponding drive.

[0027] The input components (250) include components that enable the device (200) to receive information, for example, via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). Additionally or alternatively, the input components (250) may include sensors that detect information (e.g., a Global Positioning System (GPS) component, an accelerometer, a gyroscope, and / or an actuator). The output components (260) include components that provide output information from the device (200) (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs)).

[0028] The communication interface (270) includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the device (200) to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface (270) may enable the device (200) to receive information from another device and / or provide information to another device. For example, the communication interface (270) may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a WiFi interface, a cellular network interface, etc.

[0029] The device (200) may perform one or more of the operations described herein. The device (200) may perform these operations in response to the processor (220) executing software instructions stored in a transitory computer-readable medium, such as the memory (230) and / or storage component (240). A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.

[0030] The software instructions may be loaded into the memory (230) and / or storage component (240) from another computer-readable medium or from another device via the communications interface (270). When executed, the software instructions stored in the memory (230) and / or storage component (240) may cause the processor (220) to perform one or more operations described herein. Additionally or alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to perform one or more operations described herein. Thus, the implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0031] The number and arrangement of components shown in Figure 2 are provided as an example. In practice, apparatus (200) may include additional, fewer, different, or differently arranged components than those shown in Figure 2. Additionally or alternatively, a set of components (e.g., one or more components) of apparatus (200) may perform one or more functions that are described as being performed by another set of components of apparatus (200).

[0032] The video compression framework can be described as follows: An input video x is made up of multiple image frames x 1 , ..., x Twhere T represents the total number of frames in the video. The frame may be partitioned into spatial blocks, and each block may be partitioned repeatedly into smaller blocks. Any suitable method may be used for partitioning. For example, 3D tree coding (e.g., octree partitioning) may be used. The partitioned block may include both luma and chroma components. During the intra prediction process, the luma component may be predicted first, and then the two chroma channels may be predicted later. According to an embodiment, the predictions of both chroma channels may be generated jointly or separately. The reconstructed chroma components may be generated by a DNN-based model at both the encoder and the decoder. In some embodiments, the reconstructed chroma components may be generated by a DNN-based model only at the decoder. According to embodiments, one or more processes including signal processing, spatial or temporal filtering, scaling, weighted averaging, up / down sampling, pooling, memory-based recursive processing, linear system processing, non-linear system processing, neural network processing, deep learning based processing, AI processing, pre-trained network processing, machine learning based processing, or combinations thereof may be used as modules for pre-processing of image frames and / or post-processing of image frames.

[0033] 3 is a diagram of an exemplary deep neural network (DNN) based cross-component prediction process (300) with scaling factor during encoding or decoding, according to an embodiment. As shown in FIG. 3, the process 300 may include a neural network model (302) and a reconstruction quality calculation (304).

[0034] When luma components (e.g., during encoding) or reconstructed luma components (e.g., during decoding), specific side information, or information related to neighboring luma reference blocks and neighboring chroma reference blocks are provided as input to the neural network model (302), the neural network model (302) can be trained and jointly perform estimation.

[0035] In some embodiments, the neural network model (302) may be a pre-trained model that is fine-tuned at some time before or after encoding or decoding using the neural network model (302). In some embodiments, the neural network model (302) may be pre-trained, but may be continuously updated during each encoding or decoding using estimated acceleration and continuous tuning. For continuous updates, in some embodiments, the neural network model (302) may be supported by a customized hardware processor and may be supported by a low precision floating point representation used during training.

[0036] According to an embodiment, the additional side information may include image characteristics and information provided by the encoder, including, but not limited to, luma components, block sizes, block components, quantization parameter (QP) values, and so on.

[0037] The output of the neural network model (302) may be the predicted chroma components. The two chroma channels may use different neural network based models or may use the same one. The embodiments of the present disclosure allow for combinations, and the concatenation or order of how these components are used as inputs may be changed arbitrarily.

[0038] The predicted chroma components may be used as input to a reconstruction quality computation (304) to generate a reconstructed chroma block. In some embodiments, the reconstruction quality computation (304) may use chroma blocks from other prediction modes as input. In some embodiments, the reconstruction quality computation (304) may receive an original chroma block associated with the reconstructed chroma block, determine a compression quality, and determine whether one or more parameters of the neural network model need to be or may be updated, and therefore update the neural network model.

[0039] To improve the learning speed and accuracy of the neural network-based cross-component prediction model, some additional parameters may be added to the neural network-based cross-component prediction model of the present disclosure. These one or more additional parameters may be added as learnable parameters during initial training, fine-tuning, or continuous tuning. During training, the additional parameters can be learned by optimizing a rate-distortion loss based on the input video sequence. These additional parameters are called scaling factors α. In one embodiment, the scaling factors α may be applied to weight terms or bias terms, and may share weights as weight terms or bias terms. For example, a weight tensor W may be multiplied / added by the scaling factor α to obtain a new weight tensor Ŵ. Any pattern may be applied to α. For example, if α is a 2D matrix, each row may share the same value, or each column may share the same value.

[0040] According to one embodiment, a neural network-based model for cross-component prediction may be fine-tuned or continuously updated based on a single video sequence. According to one embodiment, a neural network-based model for cross-component prediction may be fine-tuned or continuously updated based on a set of video sequences.

[0041] According to an embodiment, a model based on a neural network may be pre-trained. According to an embodiment, one or more parameters in only one layer or a certain type of layer of the neural network model may be updated and a new model is generated. In other preferred embodiments, parameters are updated in multiple or all layers of the neural network model. In an embodiment, only one or more bias terms / parameters may be optimized and updated. In an embodiment, one or more weight (coefficient) terms / parameters may be optimized and updated. In an embodiment, both one or more bias parameters and one or more weight terms / parameters may be jointly optimized or jointly optimized.

[0042] At the end of training and / or fine-tuning, updated parameters may be calculated. In an embodiment, the compression performance may be calculated between the updated parameters and the existing pre-trained parameters. In an embodiment, the updated parameters are fine-tuning parameters. That is, the neural network model may be updated with the fine-tuning parameters, replacing the existing pre-trained parameters. In another preferred embodiment, the updated parameters may be some specific transformation of the fine-tuned parameters.

[0043] According to an embodiment, data compression may be performed on the updated parameters. For example, the LZMA2 algorithm may be used to compress the updated parameters. In an embodiment, no compression may be performed.

[0044] Compared with the cross-component prediction method based on neural networks disclosed herein, the cross-component prediction method in intra prediction mode may have better compression quality. According to some embodiments, one or more parameters to be optimized may be updated based on the scaling factor, and a specific video may be used as an input for online training to improve compression performance. Furthermore, one or more scaling factors may be integrated into the model, which does not affect the estimation speed compared with previous cross-component prediction methods based on neural networks.

[0045] 4 shows a flow chart of a process 400 for deep neural network (DNN) based cross-component prediction with scaling during encoding or decoding, according to an embodiment. Process 400 may be performed using an encoder or a decoder or both.

[0046] In some embodiments, at operation 405, the pre-trained neural network model may be trained using one or more scaling factors as additional parameters, where the one or more scaling factors are determined such that one or more parameters of the pre-trained neural network model are optimized for rate-distortion loss. To In operation 405, a luma component may be received. In some examples, the luma component may already be reconstructed.

[0047] At operation 410, the chroma components may be predicted based on the luma components using a pre-trained neural network model for chroma prediction. In some embodiments, the pre-trained neural network model may be trained with one or more scaling factors as additional parameters, where the one or more scaling factors may be based on one or more parameters of the pre-trained neural network model being optimized for rate-distortion loss.

[0048] At operation 415, a reconstructed chroma component may be generated based on the predicted chroma component and one or more chroma components coded using the set of prediction modes. According to an embodiment, generating the reconstructed chroma component may be based on a quality calculation of the predicted chroma component, where the quality calculation of the predicted chroma component may be based on one or more chroma components from other prediction modes and an original chroma component associated with the predicted chroma component.

[0049] At operation 420, the pre-trained neural network model for chroma prediction may be updated using one or more scaling factors. In some embodiments, updating the pre-trained neural network model includes updating one or more scaling factors of the pre-trained neural network model based on one or more scaling factors. bias Updating the pre-trained neural network model may include optimizing parameters of one or more of the pre-trained neural network models based on one or more scaling factors. Weight Updating the pre-trained neural network model may include jointly optimizing one or more bias parameters and one or more weight parameters of the pre-trained neural network model based on the one or more scaling factors.

[0050] In some embodiments, the updating step comprises: calculating a first compression performance of the updated neural network model including the one or more parameters optimized at the one or more scaling factors; Calculating a second compression performance of the pre-trained neural network including the one or more relevant parameters; determining whether to update the pre-trained neural network model to include one or more parameters optimized by one or more scaling factors based on the comparison of the first compression performance and the second compression performance being greater than a threshold; may include:

[0051] According to some embodiments, updating the pre-trained neural network model may include optimizing one or more parameters among one or more layers of the pre-trained neural network model with one or more scaling factors. The selected one or more layers may include one or more convolutional layers, a set of final layers, or all layers of the pre-trained neural network model. According to embodiments, updating the pre-trained neural network model may include updating based on a single video sequence or a set of video sequences.

[0052] 5 shows an arrangement of a video encoder and a video decoder in a streaming environment as an example of an application of the disclosed subject matter, which is equally applicable to, for example, video conferencing, digital TV, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc., other video-enabled applications, etc.

[0053] The streaming system may include a video source (501), which may include, for example, a digital camera, generating, for example, an uncompressed video sample stream (502). The sample stream (502) may be processed by an encoder (503), shown in bold to emphasize its high data volume when compared to an encoded video bitstream, coupled to the camera (501). The encoder (503) may include hardware, software, or a combination thereof, and may enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video bitstream (504), shown in thin to emphasize its low data volume when compared to the sample stream, may be stored on a streaming server (505) for future use. One or more streaming clients (506, 508) may access the streaming server (505) to retrieve copies (507, 509) of the encoded video bitstream (504). The client (506) may include a video decoder (510). The video decoder (310) decodes an incoming copy of the encoded bitstream (507) and generates an output video sample stream (511) that can be rendered on a display (512) or other rendering device (not shown). In some streaming systems, the video bitstreams (504, 507, 509) can be encoded according to a particular video coding / compression standard. Examples of such standards include H.265 HEVC. The emerging video coding standard is known informally as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0054] FIG. 6 illustrates a block diagram of a video decoder 510 according to one embodiment of the present disclosure.

[0055] The receiver (610) may receive one or more coded video sequences to be decoded by the video decoder (610), one coded video sequence at a time in the same or another embodiment, where the decoding of each coded video sequence is independent of the other coded video sequences. The coded video sequences may be received from a channel (612), which may be a hardware / software link to a storage device that stores the coded video data. The receiver (610) may receive the coded video data along with other data, e.g., coded audio data and / or auxiliary data streams, which may be forwarded to respective usage entities (not shown). The receiver (610) may separate the coded video sequences from the other data. To eliminate network jitter, a buffer memory (615) may be connected between the receiver (610) and the entropy decoder / parser (620) (hereinafter the "parser"). When the receiver (610) is receiving data controllably from a storage / forwarding device of sufficient bandwidth or from an isosynchronous network, the buffer (615) may not be needed or may be small. For use in a best effort packet network such as the Internet, buffer (615) may be necessary and may be relatively large, and may advantageously be adaptively sized.

[0056] The video decoder (510) may include a parser (620) to reconstruct symbols (621) from the entropy coded video sequence. These symbol categories include information used to manage the operation of the decoder (510) and information for controlling a rendering device, such as a display (512), which may in some cases not be an integral part of the decoder but may be connected to the decoder as shown in FIG. 6. The control information for the rendering device may be in the form of a Supplementary Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (620) may parse / entropy decode the received code or video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without contextual dependency, etc. The parser 620 may extract a set of subgroup parameters from the coded video sequence for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the subgroup. Subgroups may include Groups of Pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The entropy decoder / parser may also extract information such as transform coefficients, quantizer parameter (QP) values, motion vectors, etc. from the coded video sequence.

[0057] The parser (620) may perform an entropy decoding / parsing operation on the video sequence received from the buffer (615) to generate symbols (621). The parser 620 may receive the encoded data and selectively decode particular symbols 621. Additionally, the parser 620 may determine whether a particular symbol 621 should be provided to a motion compensated prediction unit 653, a scaler / inverse transform unit 651, an intra prediction unit 652, or a loop filter 656.

[0058] The reconstruction of the symbols (621) may include a number of different units, depending on the type of coded video picture or portion thereof (e.g., inter and intra pictures, inter and intra blocks) and other factors. Which units are included and how can be controlled by subgroup control information parsed from the coded video sequence by the parser (620). The flow of such subgroup control information between the parser (620) and the following units is not shown for clarity.

[0059] Beyond the functional blocks already mentioned, the decoder (510) may be conceptually subdivided into a number of functional units, as described below. In an actual implementation operating under commercial constraints, many of these units will interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is adequate.

[0060] The first unit is a scalar / inverse transform unit 651. The scalar / inverse transform unit (651) receives quantized transform coefficients and control information including which transform should be used, block size, quantization coefficients, quantization scaling matrices, etc. as symbols (621) from the parser (620). It can output blocks containing sample values ​​that can be input to an aggregator (655).

[0061] In some examples, the output samples of the scaler / inverse transform (651) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information can be provided by an intra-picture prediction unit (652). In some cases, the intra-picture prediction unit (652) generates a block of the same size and shape as the block being reconstructed, using surrounding already reconstructed information fetched from the current (partially reconstructed) picture (666). The aggregator (655) adds, in some cases, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (652) to the output sample information provided by the scaler / inverse transform unit (651).

[0062] In other cases, the output samples of the scaler / inverse transform unit (651) may relate to an inter-coded, possibly motion-compensated, block. In such cases, the motion-compensated prediction unit (653) may access a reference picture memory (657) to fetch samples used for prediction. After motion-compensating the fetched samples according to the symbols (621) associated with the block, these samples may be added by an aggregator (655) to the output of the scaler / inverse transform unit to generate output sample information (in this case referred to as residual samples or residual signals). The addresses in the reference picture memory from which the motion-compensated prediction unit fetches prediction samples may be controlled by the motion-compensated prediction unit's available motion vectors, e.g., in the form of symbols (621) that may have X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory when sub-sample accurate motion vectors are in use, motion vector prediction mechanisms, etc.

[0063] The output samples of the aggregator (655) may be subjected to various loop filtering techniques in a loop filter unit (656). The video compression techniques are controlled by parameters contained in the coded video bitstream and made available to the loop filter unit (656) as symbols (621) from the parser (620), but may include in-loop filter techniques that are also responsive to meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence, and that may also be responsive to previously reconstructed loop filtered sample values.

[0064] The output of the loop filter unit (656) may be a sample stream that can be output to a render device (512) and stored in a reference picture memory (666) for use in future inter-picture prediction.

[0065] Once a particular coding picture is fully reconstructed, it can be used as a reference picture for future prediction. Once a coding picture is fully reconstructed and the coding picture is identified as a reference picture (e.g., by the parser (620)), the current reference picture (666) can become part of the reference picture buffer (657), and fresh current picture memory can be reallocated before beginning reconstruction of a subsequent coding picture.

[0066] The video decoder (510) may perform decoding operations according to a given video compression technique, which may be defined in a standard such as H.265 HEVC. The coding video sequence may comply with a syntax specified by the video compression technique or standard in use, in the sense that the coding video sequence complies with the syntax of the video compression technique or standard, as specified in the video compression technique or standard, and in particular in a profile document therein. Also, a requirement for compliance may be that the complexity of the coding video sequence is within the limits defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples / second), maximum reference picture size, etc. The limits set by the level may in some cases be further limited through a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the coding video sequence.

[0067] In an embodiment, the receiver (610) may receive additional (redundant) data along with the coded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (510) to correctly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0068] FIG. 7 may be a functional block diagram of a video encoder (503) according to one embodiment of the disclosure.

[0069] The encoder (503) may receive video samples from a video source (501) (not part of the encoder) that may capture video images to be coded by the encoder (503).

[0070] The video source (501) may provide a source video sequence to be coded by the encoder (503) in the form of a digital video sample stream of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCb, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media presentation system, the video source (501) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (503) may be a camera that captures local image information as a video sequence. The video data may be provided as a number of individual pictures that, when viewed in succession, give the appearance of motion. The pictures themselves may be organized as a spatial array of pixels. Each pixel may contain one or more samples, depending on the sampling structure, color space, etc., being used. Those skilled in the art will readily appreciate the relationship between pixels and samples. The following description focuses on samples.

[0071] According to one embodiment, the encoder (503) may code and compress pictures of a source video sequence into a coded video sequence (743) in real-time or under any other time constraint required by the application. Enforcing an appropriate coding rate is one function of the controller (750). The controller (750) may control other functional units as described below and is operatively coupled to these units. Coupling is not shown for clarity. Parameters set by the controller may include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art will readily identify other functions of the controller (750) as they may relate to an optimized video encoder (503) for a particular system design.

[0072] Some video encoders operate in what those skilled in the art would immediately recognize as a "coding loop." As a very simplified explanation, the coding loop can include an encoder (730) (hereafter "source coder") (which generates symbols based on an input picture to be coded and reference pictures) and a coding portion of a (local) decoder (733) that is embedded within the encoder (503) and reconstructs the symbols to generate sample data that a (remote) decoder can generate (when any compression between the symbols and the coding video bitstream is lossless among the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream is input to a reference picture memory 734. When the decoding of the symbol stream results in bit-exact results independent of the decoder location (local or remote), the contents of the reference picture buffer are also bit-exact between the local and remote encoders. In other words, the predictive portion of the encoder "sees" exactly the same sample values ​​as the decoder "sees" when using prediction during decoding as reference picture samples. This basic principle of reference picture synchrony (and the resulting drift if synchrony cannot be maintained, for example due to channel errors) is well known to those skilled in the art.

[0073] The operation of the "local" decoder (733) may be the same as that of the "remote" decoder (510), detailed above in connection with Figure 16. Referring also briefly to Figure 16, however, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (745) and parser (620) may be lossless, the entropy decoding portion of the decoder (510), including the channel (612), receiver (610), buffer (615), and parser (620), may not be fully implemented in the local decoder (733).

[0074] An observation to be made at this point is that any decoder techniques, other than parsing / entropy decoding, present in the decoder must also be present in substantially the same functional form as in the corresponding encoder. Descriptions of the encoder techniques can be omitted since they are the inverse of the decoder techniques which are described generically. Only in certain areas are more detailed descriptions necessary and are provided below.

[0075] In operation, in some examples, the source coder (730) may perform motion-compensated predictive coding, which predictively codes an input frame with reference to one or more previously coded frames from the video sequence designated as "reference frames." In this method, the coding engine (732) codes differences between pixel blocks of the input frame and pixel blocks of reference frames, which may be selected as prediction references for the input frame.

[0076] The local video decoder (733) may decode the coded video data of frames, which may be designated as reference frames, based on symbols generated by the source coder (730). The operation of the coding engine (732) may advantageously be lossy. When the coded video data may be decoded in a video decoder (not shown in FIG. 17), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (733) may replicate the decoding process that may be performed by the video decoder on the reference frames, resulting in reconstructed reference frames to be stored in the reference picture cache (734). In this way, the encoder (503) may locally store copies of reconstructed reference frames that have common content with the reconstructed reference frames that would be obtained by the far-end video decoder (in the absence of transmission errors).

[0077] The predictor (735) may perform a prediction search for the coding engine (732). That is, for a new frame to be coded, the predictor (735) may search the reference picture memory (734) for sample data (such as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc. that may serve as suitable prediction references for the new picture. The predictor (735) may operate on a sample block-pixel block basis to find a suitable prediction reference. In some examples, an input picture may have prediction references derived from multiple reference pictures stored in the reference picture memory (734), as determined by the search results obtained by the predictor (735).

[0078] The controller (750) may manage the coding operations of the video coder (730), including, for example, setting parameters and subgroup parameters used for encoding the video data.

[0079] The output of all the aforementioned functional units may undergo entropy coding in an entropy coder (745), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques well known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, etc.

[0080] The transmitter (740) may buffer the coded video sequence generated by the entropy coder (745) to prepare it for transmission over a communication channel (760), which may be a hardware / software link to a storage device that may store the coded video data. The transmitter (740) may merge the coded video data from the video coder (730) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0081] A control unit (750) may manage the operation of the encoder (503). During coding, the control unit (750) may assign a particular coding picture type to each coding picture, which may affect the coding technique that may be applied to each picture. For example, pictures may often be assigned as one of the following picture types:

[0082] An intra picture (I-picture) may be a picture that can be coded and decoded without using any other frame in a sequence as a source of prediction. Some video codecs allow different kinds of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will recognize the variations of I-pictures and their respective applications and characteristics.

[0083] A predictive picture (P-picture) may be a picture that can be coded and decoded using intra- or inter-prediction, in most cases using one motion vector and reference index to predict the sample values ​​of each block.

[0084] A Bi-directionally Predictive Picture (B Picture) may be a picture that can be coded and decoded using intra- or inter-prediction, using up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, a multi-predictive picture can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0085] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks determined by the coding assignment applied to each picture of the block. For example, blocks of I pictures may be non-predictively coded, or they may be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of P pictures may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. Blocks of B pictures may be non-predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.

[0086] The video coder (503) may perform coding operations according to a given video coding technique or standard, such as H.265 HEVC, in which the video coder (503) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. The coding video data may therefore conform to a syntax specified by the video coding technique or standard being used.

[0087] In one embodiment, the transmitter (740) may transmit additional data along with the encoded video. The video coder (730) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0088] This disclosure is directed to several block partitioning methods in which motion information is taken into account during tree partitioning for video coding. More specifically, the techniques of this disclosure relate to tree partitioning methods for flexible tree structures based on motion field information. The techniques proposed in this disclosure may be applied to both homogeneous and heterogeneous derived motion fields.

[0089] The derived motion field of a block is defined to be homogeneous if the derived motion field is available for all sub-blocks in the block and all motion vectors in the derived motion field are similar, e.g., if the motion vectors share the same reference frame and the absolute differences between the motion vectors are all below a certain threshold, which may be signaled in the bitstream or may be predefined.

[0090] The derived motion field of a block is defined to be heterogeneous if the derived motion field is available for all sub-blocks within the block and the motion vectors within the derived motion field are not similar, e.g., if at least one motion vector references a reference frame that is not referenced by the other motion vector, or if at least one absolute difference between two motion vectors within the field is greater than a signaled or predefined threshold.

[0091] While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents, which are encompassed within the scope of this disclosure. It will be apparent to those skilled in the art that numerous systems and methods can be devised that, although not explicitly shown or described herein, embody the principles of the disclosure and thus are encompassed within the spirit and scope of the disclosure.

Claims

1. 1. A method for scaling factor based neural network (NN) cross-component prediction during decoding, the method being executed by one or more processors, the method comprising: training a deep neural network (DNN) cross-component prediction (CCP) model with at least one or more scaling factors, the at least one or more scaling factors being learned by optimizing a rate-distortion loss for one or more parameters of the DNN CCP model based on an input video sequence including a luma component, and applying one of the at least one or more scaling factors to one of the one or more parameters; Reconstructing chroma components based on the luma component using the trained DNN CCP model with a learned scaling factor for chroma prediction; updating the trained DNN CCP model for chroma prediction of the input video sequence using the learned scaling factors; performing chroma prediction of the input video sequence using the updated DNN CCP model at the learned scaling factor; The method includes:

2. 2. The method of claim 1, wherein updating the trained DNN CCP model comprises optimizing one or more bias parameters of the trained DNN CCP model based on the learned scaling factors.

3. 2. The method of claim 1, wherein updating the trained DNN CCP model comprises optimizing one or more weight parameters of the trained DNN CCP model based on the learned scaling factors.

4. 2. The method of claim 1, wherein updating the trained DNN CCP model comprises jointly optimizing one or more bias parameters and one or more weight parameters of the trained DNN CCP model based on the learned scaling factors.

5. 2. The method of claim 1, wherein updating the trained DNN CCP model comprises optimizing one or more parameters in one or more layers of the trained DNN CCP model based on the learned scaling factors.

6. 6. The method of claim 5, wherein the one or more layers include one or more convolutional layers of the trained DNN CCP model.

7. The method of claim 5 , wherein the one or more layers comprise a final set of layers of the trained DNN CCP model.

8. 6. The method of claim 5, wherein the one or more layers include all layers of the trained DNN CCP model having the same layer properties.

9. 2. The method of claim 1, wherein updating the trained DNN CCP model further comprises updating based on a single video sequence or the set of video sequences.

10. The method of claim 1 , wherein the quality calculation of the reconstructed chroma component is based on one or more chroma components from other prediction modes and an original chroma component associated with the reconstructed chroma component.

11. The update steps are: calculating a first compression performance of an updated neural network model including one or more parameters optimized with the learned scaling factor; Calculating a second compression performance of the trained DNN CCP model including the one or more associated parameters; determining whether to update the trained DNN CCP model to include one or more parameters optimized by the learned scaling factor based on a comparison of the first compression performance and the second compression performance being greater than a threshold; The method of claim 1 , comprising:

12. Apparatus for neural network (NN) based cross-component prediction with scaling factor during decoding, said apparatus executing the method according to any one of claims 1 to 11.

13. 12. A computer program for scaling factor-based neural network (NN) cross-component prediction during decoding, the computer program being configured, when executed by at least one processor, to cause the at least one processor to perform the method of any one of claims 1 to 11.

14. 1. A method for scaling factor based neural network (NN) cross-component prediction during encoding, the method being executed by one or more processors, the method comprising: training a deep neural network (DNN) cross-component prediction (CCP) model with at least one or more scaling factors, the at least one or more scaling factors being learned by optimizing a rate-distortion loss for one or more parameters of the DNN CCP model based on an input video sequence including a luma component, and applying one of the at least one or more scaling factors to one of the one or more parameters; reconstructing chroma components based on the luma component using the trained DNN CCP model with the learned scaling factor for chroma prediction; updating the trained DNN CCP model for chroma prediction of the input video sequence using the learned scaling factors; performing chroma prediction of the input video sequence using the updated DNN CCP with the learned scaling factor; The method includes:

Images